Jul 02 21:18:49.785762 ip-10-0-130-121 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.124601    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128447    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128473    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128478    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128481    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128484    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128487    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128490    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:50.207205 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128493    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:50.193750 ip-10-0-130-121 systemd[1]: Started Kubernetes Kubelet.
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128496    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128499    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128502    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128504    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128507    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128514    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128517    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128519    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128522    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128525    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128527    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128530    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128533    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128535    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128538    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128540    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128543    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128546    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128549    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:50.208689 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128552    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128555    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128558    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128561    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128564    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128566    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128569    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128571    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128574    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128576    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128579    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128581    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128584    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128586    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.128589    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129191    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129196    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129200    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129203    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:50.210348 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129206    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129209    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129211    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129214    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129217    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129219    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129222    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129224    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129227    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129230    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129233    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129235    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129238    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129240    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129243    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129246    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129249    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129251    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129253    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129256    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:50.211369 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129259    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129262    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129264    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129267    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129270    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129273    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129275    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129280    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129285    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129288    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129292    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129296    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129300    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129304    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129307    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129310    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129312    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129316    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:50.212252 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129320    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129322    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129325    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129328    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129331    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129334    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129337    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129340    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129342    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129345    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129348    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129350    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129353    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129356    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.129358    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131228    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131235    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131238    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131241    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131244    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:50.213181 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131247    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131250    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131253    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131256    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131258    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131261    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131264    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131267    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131269    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131272    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131274    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131277    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131279    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131282    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131284    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131287    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131290    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131292    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131295    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131297    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:50.214329 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131300    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131302    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131305    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131308    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131311    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131314    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131316    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131319    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131321    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131324    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131328    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131331    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131334    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131337    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131340    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131343    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131346    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131348    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131352    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:50.216272 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131355    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131358    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131361    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131364    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131366    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131369    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131371    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131374    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131378    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131380    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131383    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131386    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131390    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131392    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131395    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131398    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131400    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131403    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131407    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:50.217318 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131411    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131414    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131417    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131420    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131423    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131426    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131429    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131432    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131435    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131437    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131440    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131443    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131445    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131448    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131451    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131468    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131471    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131474    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131477    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131481    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:50.218141 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131483    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131486    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131489    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131491    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131495    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131498    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131501    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131504    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131506    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131509    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131512    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131514    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131517    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131522    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131526    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131530    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131968    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131973    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131976    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131979    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:50.218751 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131982    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131985    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131988    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131991    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131994    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.131998    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132001    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132004    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132007    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132010    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132012    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132015    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132017    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132020    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132023    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132025    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132029    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132031    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132034    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:50.219987 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132037    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132039    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132042    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132044    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132047    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132050    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132055    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132057    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132060    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132062    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132066    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132070    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132073    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132076    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132079    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132082    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132085    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132088    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132090    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:50.220612 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132093    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132096    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132098    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132101    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132103    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132106    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132109    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132111    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132114    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132116    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132119    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132121    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132124    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132127    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132129    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132132    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132135    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132137    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132140    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132145    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:50.221862 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132147    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132151    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132155    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132157    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132160    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132162    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132165    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132167    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132172    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132175    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132179    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132182    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132184    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132187    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132190    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132193    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132195    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132198    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132200    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132204    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:50.222726 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132207    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132210    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132212    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132215    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132217    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132220    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132222    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132225    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132227    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132230    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132232    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132236    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132238    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132241    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132244    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132247    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.132249    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132322    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132332    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132342    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132349    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 21:18:50.223363 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132358    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132364    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132374    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132386    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132396    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132402    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132408    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132416    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132423    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132429    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132435    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132439    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132442    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132445    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132449    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132452    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132467    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132470    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132474    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132478    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132481    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132484    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132489    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132492    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 21:18:50.224242 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132495    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132498    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132501    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132504    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132508    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132511    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132514    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132517    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132520    2564 flags.go:64] FLAG: --enable-server="true"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132523    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132528    2564 flags.go:64] FLAG: --event-burst="100"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132531    2564 flags.go:64] FLAG: --event-qps="50"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132534    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132537    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132540    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132544    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132547    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132550    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132555    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132558    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132561    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132563    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132566    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132569    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132572    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 21:18:50.225344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132575    2564 flags.go:64] FLAG: --feature-gates=""
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132579    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132582    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132585    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132588    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132592    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132596    2564 flags.go:64] FLAG: --help="false"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132599    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-130-121.ec2.internal"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132602    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132605    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132608    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132611    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132615    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132618    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132620    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132623    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132626    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132629    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132632    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132635    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132638    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132640    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132643    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132647    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 21:18:50.226422 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132650    2564 flags.go:64] FLAG: --lock-file=""
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132654    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132657    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132659    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132665    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132668    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132670    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132673    2564 flags.go:64] FLAG: --logging-format="text"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132676    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132679    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132682    2564 flags.go:64] FLAG: --manifest-url=""
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132685    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132689    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132693    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132697    2564 flags.go:64] FLAG: --max-pods="110"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132700    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132703    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132706    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132709    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132712    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132715    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132718    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132726    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132729    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132732    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 21:18:50.227408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132735    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132738    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132743    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132746    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132749    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132752    2564 flags.go:64] FLAG: --port="10250"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132755    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132763    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04e3c0a34a687d8c0"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132767    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132770    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132774    2564 flags.go:64] FLAG: --register-node="true"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132777    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132780    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132783    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132786    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132789    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132793    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132796    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132799    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132802    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132804    2564 flags.go:64] FLAG: --runonce="false"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132808    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132812    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132814    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132817    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 21:18:50.228388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132820    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132823    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132826    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132829    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132832    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132835    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132838    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132841    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132844    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132846    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132849    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132855    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132857    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132860    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132864    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132869    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132872    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132875    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132878    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132881    2564 flags.go:64] FLAG: --v="2"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132885    2564 flags.go:64] FLAG: --version="false"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132889    2564 flags.go:64] FLAG: --vmodule=""
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132892    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.132896    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133013    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:50.229335 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133017    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133020    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133023    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133027    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133029    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133032    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133035    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133037    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133040    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133042    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133045    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133047    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133050    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133053    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133055    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133058    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133060    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133063    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133066    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133068    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:50.230262 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133071    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133074    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133077    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133080    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133083    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133086    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133088    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133091    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133093    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133096    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133099    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133101    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133104    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133106    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133109    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133113    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133116    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133118    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133121    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133124    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:50.230909 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133126    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133129    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133131    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133135    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133139    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133142    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133145    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133148    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133150    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133153    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133156    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133158    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133161    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133165    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133170    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133174    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133177    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133180    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133183    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:50.231777 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133185    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133189    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133192    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133195    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133198    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133200    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133203    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133206    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133210    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133212    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133215    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133217    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133220    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133222    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133226    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133230    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133233    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133235    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133238    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:50.233107 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133240    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133243    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133245    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133248    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133251    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133253    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133256    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133259    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133263    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133265    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133268    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133271    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133273    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133276    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133279    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133281    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133284    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133286    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133289    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:50.299242 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.133291    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.133830    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.141117    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.141135    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.141150    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.141158    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141232    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141237    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141241    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141244    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141247    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141250    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141252    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141255    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141258    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:50.300259 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141260    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141263    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141265    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141268    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141270    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141273    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141280    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141284    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141287    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141290    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141294    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141297    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141300    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141303    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141305    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141308    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141311    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141314    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141316    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:50.300863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141319    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141328    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141331    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141334    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141336    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141340    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141342    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141345    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141347    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141350    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141353    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141355    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141358    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141361    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141363    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141366    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141368    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141371    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141376    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:50.301597 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141380    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141382    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141385    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141387    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141390    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141392    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141395    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141397    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141400    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141403    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141405    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141408    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141410    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141413    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141415    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141423    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141427    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141431    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141435    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141439    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:50.302283 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141443    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141447    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141451    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141454    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141475    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141479    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141483    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141487    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141490    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141493    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141497    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141501    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141506    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141509    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141512    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141514    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141517    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141519    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141522    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:50.302988 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141524    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141527    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141530    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141532    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141544    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141546    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141549    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141552    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141556    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141567    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141569    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141572    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141575    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.141580    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141739    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 21:18:50.303682 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141745    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141747    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141750    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141753    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141755    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141758    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141760    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141763    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141765    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141769    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141772    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141777    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141781    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141784    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141787    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141790    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141793    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141795    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141798    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 21:18:50.304213 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141800    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141803    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141805    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141808    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141810    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141813    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141816    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141818    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141826    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141829    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141831    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141834    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141836    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141839    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141842    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141844    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141847    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141849    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141852    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141854    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 21:18:50.304804 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141857    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141859    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141863    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141866    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141870    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141872    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141875    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141877    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141880    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141883    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141885    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141888    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141891    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141893    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141896    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141899    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141901    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141904    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141906    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141909    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 21:18:50.305402 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141912    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141920    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141922    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141925    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141927    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141930    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141932    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141935    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141938    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141940    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141943    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141945    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141948    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141950    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141955    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141957    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141961    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141964    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141966    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141969    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 21:18:50.306066 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141971    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141974    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141976    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141980    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141983    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141987    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141990    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141993    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141996    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.141999    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.142002    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.142005    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.142008    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.142010    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.142020    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.142023    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.142025    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.142028    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 21:18:50.306754 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:50.142030    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.142035    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.142181    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.144310    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.145563    2564 server.go:1019] "Starting client certificate rotation"
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.147023    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.147686    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.150344    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.151493    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.159968    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.162023    2564 log.go:25] "Validated CRI v1 image API"
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.162811    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 21:18:50.307409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.167034    2564 fs.go:135] Filesystem UUIDs: map[3c449b2d-e28c-4f9b-9192-641915dfa557:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 93ceb27b-964a-43db-af46-eb98e37fa971:/dev/nvme0n1p3]
Jul 02 21:18:50.307854 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.167049    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 21:18:50.308071 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.172367    2564 manager.go:216] Machine: {Timestamp:2026-07-02 21:18:50.171309802 +0000 UTC m=+0.293478205 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101445 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2264702752e54255bfb0074ea7f9f3 SystemUUID:ec226470-2752-e542-55bf-b0074ea7f9f3 BootID:150c1e4f-d487-4592-b899-f1d912dfd9a0 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:72:00:8f:79:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:72:00:8f:79:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:69:77:88:2f:36 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 21:18:50.308071 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.172479    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 21:18:50.308071 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.172590    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 21:18:50.308071 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.174961    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 21:18:50.308071 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.174987    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 21:18:50.308071 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.175547    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 21:18:50.308071 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.175557    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 21:18:50.308071 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.175565    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.175578    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.176666    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.180351    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.180482    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.181015    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.181024    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.181038    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.181047    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.181056    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.181375    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.181392    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.184514    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.186803    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.186840    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188556    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188582    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188596    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188603    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188612    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188618    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188627    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188633    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188640    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188646    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188655    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188664    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188697    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.188705    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.192407    2564 server.go:1268] "Started kubelet"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.192494    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.192536    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.192755    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.196763    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.196983    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.198366    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.198994    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.199085    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.199091    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 21:18:50.308715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.199883    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.201849    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.201864    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.201906    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.202381    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.202384    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.202403    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.202575    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.202584    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.202901    2564 factory.go:55] Registering systemd factory
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.202912    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.203062    2564 factory.go:153] Registering CRI-O factory
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.203069    2564 factory.go:223] Registration of the crio container factory successfully
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.203109    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.203127    2564 factory.go:103] Registering Raw factory
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.203137    2564 manager.go:1196] Started watching for new ooms in manager
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.203403    2564 manager.go:349] Starting recovery of all containers
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.211527    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.211596    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.211682    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-121.ec2.internal.18be95ba60003ec6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-121.ec2.internal,UID:ip-10-0-130-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-121.ec2.internal,},FirstTimestamp:2026-07-02 21:18:50.19237959 +0000 UTC m=+0.314547972,LastTimestamp:2026-07-02 21:18:50.19237959 +0000 UTC m=+0.314547972,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-121.ec2.internal,}"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.213334    2564 manager.go:354] Recovery completed
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.223955    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.226388    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.226430    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.226444    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.227110    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.227121    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.227130    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.227140    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.228049    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.228085    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:50.309982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.228096    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.228500    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-121.ec2.internal.18be95ba6207720b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-121.ec2.internal,UID:ip-10-0-130-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-121.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-121.ec2.internal,},FirstTimestamp:2026-07-02 21:18:50.226405899 +0000 UTC m=+0.348574285,LastTimestamp:2026-07-02 21:18:50.226405899 +0000 UTC m=+0.348574285,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-121.ec2.internal,}"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.229423    2564 policy_none.go:49] "None policy: Start"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.229435    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.229445    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.230505    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tp7gh"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.233492    2564 policy_none.go:47] "Start"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.236852    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tp7gh"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.238686    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-121.ec2.internal.18be95ba6207ea84  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-121.ec2.internal,UID:ip-10-0-130-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-121.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-121.ec2.internal,},FirstTimestamp:2026-07-02 21:18:50.22643674 +0000 UTC m=+0.348605123,LastTimestamp:2026-07-02 21:18:50.22643674 +0000 UTC m=+0.348605123,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-121.ec2.internal,}"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.288112    2564 manager.go:337] "Starting Device Plugin manager"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.288340    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.288355    2564 server.go:87] "Starting device plugin registration server"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.288662    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.288675    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.288773    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.288867    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:50.288877    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.289394    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 21:18:50.312699 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:50.289449    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:51.813141 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.777841    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:51.813141 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.785006    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:51.813141 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.785060    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:51.813141 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.785074    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:51.813141 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.785134    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-121.ec2.internal"
Jul 02 21:18:51.813141 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.793109    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-121.ec2.internal"
Jul 02 21:18:51.813141 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:51.793130    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-121.ec2.internal\": node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:51.834953 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:51.817253    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:51.834953 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.826670    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 21:18:51.834953 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.828044    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 21:18:51.834953 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.828081    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 21:18:51.834953 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.828103    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 21:18:51.834953 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:51.828134    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 21:18:51.834953 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.831573    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:51.939051 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:51.917579    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:51.939051 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.928728    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-121.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal"]
Jul 02 21:18:51.939051 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.928802    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:51.939051 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.938829    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:51.939051 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.938863    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:51.939051 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.938876    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.941170    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.941276    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.941304    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.941957    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.941983    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.941956    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.942037    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.942051    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.942013    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.944341    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.944364    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.945068    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.945090    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.945099    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:51.991895    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-121.ec2.internal\" not found" node="ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.993977    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fddbb2f027add0a069d4010707bb8cdd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-121.ec2.internal\" (UID: \"fddbb2f027add0a069d4010707bb8cdd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.994047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3a57b42eebfe2a79b8f7b7b30c86f49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal\" (UID: \"e3a57b42eebfe2a79b8f7b7b30c86f49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:51.994086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3a57b42eebfe2a79b8f7b7b30c86f49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal\" (UID: \"e3a57b42eebfe2a79b8f7b7b30c86f49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:51.996529    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-121.ec2.internal\" not found" node="ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.053341 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:52.017926    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:52.160871 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.094649    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3a57b42eebfe2a79b8f7b7b30c86f49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal\" (UID: \"e3a57b42eebfe2a79b8f7b7b30c86f49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.160871 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.094691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fddbb2f027add0a069d4010707bb8cdd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-121.ec2.internal\" (UID: \"fddbb2f027add0a069d4010707bb8cdd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.160871 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.094712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3a57b42eebfe2a79b8f7b7b30c86f49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal\" (UID: \"e3a57b42eebfe2a79b8f7b7b30c86f49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.160871 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.094763    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3a57b42eebfe2a79b8f7b7b30c86f49-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal\" (UID: \"e3a57b42eebfe2a79b8f7b7b30c86f49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.160871 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.094767    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3a57b42eebfe2a79b8f7b7b30c86f49-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal\" (UID: \"e3a57b42eebfe2a79b8f7b7b30c86f49\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.160871 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.094827    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fddbb2f027add0a069d4010707bb8cdd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-121.ec2.internal\" (UID: \"fddbb2f027add0a069d4010707bb8cdd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.160871 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:52.118743    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:52.306016 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:52.219270    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:52.306016 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.294651    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.306016 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.300368    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.467355 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:52.320155    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:52.467355 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:52.420556    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:52.467355 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.455898    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:52.503901 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.503875    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:52.520977 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:52.520953    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:52.534135 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.534108    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 21:18:52.534247 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.534233    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 21:18:52.534307 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.534267    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 21:18:52.534307 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.534286    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 21:18:52.621206 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:52.621130    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:52.628297 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.628269    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 21:13:51 +0000 UTC" deadline="2028-02-14 10:28:53.870992112 +0000 UTC"
Jul 02 21:18:52.628297 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.628293    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14197h10m1.242701725s"
Jul 02 21:18:52.721931 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:52.721893    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:52.775061 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.775041    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 21:18:52.776500 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.776483    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 21:18:52.776600 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.776516    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 21:18:52.776600 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.776532    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeHasSufficientPID"
Jul 02 21:18:52.822818 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:52.822786    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:52.892572 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:52.892532    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a57b42eebfe2a79b8f7b7b30c86f49.slice/crio-a706621ec1805701ee12db3010c6857d4e464d887fc34ddf75820306aad95e88 WatchSource:0}: Error finding container a706621ec1805701ee12db3010c6857d4e464d887fc34ddf75820306aad95e88: Status 404 returned error can't find the container with id a706621ec1805701ee12db3010c6857d4e464d887fc34ddf75820306aad95e88
Jul 02 21:18:52.893023 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:52.892981    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddbb2f027add0a069d4010707bb8cdd.slice/crio-aedf9ee2af6bb31b382c678a48358beb42088542d766136b52b38fc1f265182c WatchSource:0}: Error finding container aedf9ee2af6bb31b382c678a48358beb42088542d766136b52b38fc1f265182c: Status 404 returned error can't find the container with id aedf9ee2af6bb31b382c678a48358beb42088542d766136b52b38fc1f265182c
Jul 02 21:18:52.898377 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.898363    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:18:52.922933 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:52.922908    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-121.ec2.internal\" not found"
Jul 02 21:18:52.961841 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.961813    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:52.991063 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.991026    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:52.998762 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.998743    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 21:18:52.999497 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:52.999482    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal"
Jul 02 21:18:53.009678 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.009657    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 21:18:53.570479 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.570451    2564 apiserver.go:52] "Watching apiserver"
Jul 02 21:18:53.579544 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.579512    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 21:18:53.580480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.580051    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-cd8lj","kube-system/global-pull-secret-syncer-mdqbk","kube-system/kube-apiserver-proxy-ip-10-0-130-121.ec2.internal","openshift-image-registry/node-ca-g55b4","openshift-multus/multus-additional-cni-plugins-wdmdj","openshift-multus/multus-bllj5","openshift-network-diagnostics/network-check-target-pjcpf","kube-system/konnectivity-agent-nm8zf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7","openshift-cluster-node-tuning-operator/tuned-4lmlv","openshift-dns/node-resolver-6st8d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal","openshift-multus/network-metrics-daemon-fd5sb","openshift-network-operator/iptables-alerter-gw9bj"]
Jul 02 21:18:53.582775 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.582753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.584937 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.584872    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:53.585053 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:53.584951    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:18:53.585593 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.585420    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vb7wc\""
Jul 02 21:18:53.585593 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.585463    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:18:53.585593 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.585420    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 21:18:53.588479 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.588095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gw9bj"
Jul 02 21:18:53.589488 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.589419    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g55b4"
Jul 02 21:18:53.590827 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.590807    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 21:18:53.591053 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.591037    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 21:18:53.591247 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.591231    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 21:18:53.591493 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.591476    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 21:18:53.591569 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.591509    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d4pz6\""
Jul 02 21:18:53.595109 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.593115    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.595109 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.594608    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 21:18:53.595109 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.594764    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 21:18:53.595109 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.594839    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 21:18:53.595355 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.595157    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hfw7r\""
Jul 02 21:18:53.596058 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.596037    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 21:18:53.596873 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.596845    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j9jmq\""
Jul 02 21:18:53.596980 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.596968    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 21:18:53.597112 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.597094    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 21:18:53.598663 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.597327    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 21:18:53.598663 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.597371    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 21:18:53.598663 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.597842    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.600336 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.600312    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.601344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.601320    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 21:18:53.601480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.601463    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nmg2z\""
Jul 02 21:18:53.602874 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.602852    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-szc9x\""
Jul 02 21:18:53.603038 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.602987    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 21:18:53.603116 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603047    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 21:18:53.603713 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603458    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:18:53.603713 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:53.603521    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:18:53.603851 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-sysconfig\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.603851 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603796    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-systemd\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.603851 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-run\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.603851 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603846    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhpz\" (UniqueName: \"kubernetes.io/projected/a1618b15-d851-4f6d-a6af-d6e132341463-kube-api-access-8jhpz\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.604081 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603872    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhhvk\" (UniqueName: \"kubernetes.io/projected/178a3739-7dec-45fc-a1f0-58d7e82806ff-kube-api-access-mhhvk\") pod \"iptables-alerter-gw9bj\" (UID: \"178a3739-7dec-45fc-a1f0-58d7e82806ff\") " pod="openshift-network-operator/iptables-alerter-gw9bj"
Jul 02 21:18:53.604081 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603894    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1c5ca752-f1cb-483c-9f16-7b425d081d7b-serviceca\") pod \"node-ca-g55b4\" (UID: \"1c5ca752-f1cb-483c-9f16-7b425d081d7b\") " pod="openshift-image-registry/node-ca-g55b4"
Jul 02 21:18:53.604081 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603916    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-var-lib-kubelet\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.604081 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603938    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-host\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.604081 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1618b15-d851-4f6d-a6af-d6e132341463-tmp\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.604081 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.603982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1618b15-d851-4f6d-a6af-d6e132341463-etc-tuned\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.604081 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:53.604081 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/178a3739-7dec-45fc-a1f0-58d7e82806ff-host-slash\") pod \"iptables-alerter-gw9bj\" (UID: \"178a3739-7dec-45fc-a1f0-58d7e82806ff\") " pod="openshift-network-operator/iptables-alerter-gw9bj"
Jul 02 21:18:53.604081 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604067    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgvb\" (UniqueName: \"kubernetes.io/projected/1c5ca752-f1cb-483c-9f16-7b425d081d7b-kube-api-access-8kgvb\") pod \"node-ca-g55b4\" (UID: \"1c5ca752-f1cb-483c-9f16-7b425d081d7b\") " pod="openshift-image-registry/node-ca-g55b4"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-system-cni-dir\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604116    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-modprobe-d\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604139    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-kubernetes\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604161    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-os-release\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604183    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-sysctl-d\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c5ca752-f1cb-483c-9f16-7b425d081d7b-host\") pod \"node-ca-g55b4\" (UID: \"1c5ca752-f1cb-483c-9f16-7b425d081d7b\") " pod="openshift-image-registry/node-ca-g55b4"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604246    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5cst\" (UniqueName: \"kubernetes.io/projected/30cc5577-2894-4468-9c39-a462c823e44e-kube-api-access-z5cst\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604116    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604337    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604369    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/247b8f4d-bbc5-44b7-981b-aa42c30c031f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604394    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/247b8f4d-bbc5-44b7-981b-aa42c30c031f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604424    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fdjd\" (UniqueName: \"kubernetes.io/projected/247b8f4d-bbc5-44b7-981b-aa42c30c031f-kube-api-access-2fdjd\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604448    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-sysctl-conf\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.604625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/178a3739-7dec-45fc-a1f0-58d7e82806ff-iptables-alerter-script\") pod \"iptables-alerter-gw9bj\" (UID: \"178a3739-7dec-45fc-a1f0-58d7e82806ff\") " pod="openshift-network-operator/iptables-alerter-gw9bj"
Jul 02 21:18:53.605309 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604514    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-sys\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.605309 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604538    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-cnibin\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.605309 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604567    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/247b8f4d-bbc5-44b7-981b-aa42c30c031f-cni-binary-copy\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.605309 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.604611    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-lib-modules\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.605788 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.605753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:53.605858 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:53.605819    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:18:53.608883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.608865    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.610931 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.610912    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6st8d"
Jul 02 21:18:53.612741 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.612724    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 21:18:53.612922 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.612874    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 21:18:53.613074 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.612762    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 21:18:53.613278 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.613253    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:18:53.613463 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.613445    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 21:18:53.613542 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.612811    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jgkd9\""
Jul 02 21:18:53.613635 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.613083    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 21:18:53.613719 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.613701    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 21:18:53.613891 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.613873    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 21:18:53.614294 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.614277    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 21:18:53.614434 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.614418    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tczz5\""
Jul 02 21:18:53.615411 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.615385    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 21:18:53.616165 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.616145    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 21:18:53.616375 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.616359    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 21:18:53.616448 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.616363    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jsx94\""
Jul 02 21:18:53.632024 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.631985    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g2g2j"
Jul 02 21:18:53.642661 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.642638    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g2g2j"
Jul 02 21:18:53.692469 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.692443    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 21:18:53.694786 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.694759    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:53.705556 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705531    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-modprobe-d\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.705556 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-kubernetes\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.705732 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705586    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-run-systemd\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.705732 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705610    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-node-log\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.705732 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-cni-dir\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.705732 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705654    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-etc-openvswitch\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.705732 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705667    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-kubernetes\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.705732 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705679    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/247b8f4d-bbc5-44b7-981b-aa42c30c031f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.705732 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fdjd\" (UniqueName: \"kubernetes.io/projected/247b8f4d-bbc5-44b7-981b-aa42c30c031f-kube-api-access-2fdjd\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.705732 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-sys\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705754    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-lib-modules\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-kubelet\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705797    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-cni-netd\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705820    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-system-cni-dir\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705841    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zltnq\" (UniqueName: \"kubernetes.io/projected/6c7ccba7-ce5b-45ac-ba36-22db25a36607-kube-api-access-zltnq\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/178a3739-7dec-45fc-a1f0-58d7e82806ff-iptables-alerter-script\") pod \"iptables-alerter-gw9bj\" (UID: \"178a3739-7dec-45fc-a1f0-58d7e82806ff\") " pod="openshift-network-operator/iptables-alerter-gw9bj"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705897    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-cnibin\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/247b8f4d-bbc5-44b7-981b-aa42c30c031f-cni-binary-copy\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705945    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d928b5bc-8033-4f03-be65-0989dabe8581-hosts-file\") pod \"node-resolver-6st8d\" (UID: \"d928b5bc-8033-4f03-be65-0989dabe8581\") " pod="openshift-dns/node-resolver-6st8d"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2mlr\" (UniqueName: \"kubernetes.io/projected/61608ec6-2071-4478-9bbf-4f8a55085fd9-kube-api-access-r2mlr\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706013    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-socket-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.705705    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-modprobe-d\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706050    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71429679-06c0-480b-8c79-08a2c5682ca1-kubelet-config\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:53.706137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706118    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-cnibin\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706152    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-sys\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706198    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-lib-modules\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-sysconfig\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706258    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhpz\" (UniqueName: \"kubernetes.io/projected/a1618b15-d851-4f6d-a6af-d6e132341463-kube-api-access-8jhpz\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706286    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhhvk\" (UniqueName: \"kubernetes.io/projected/178a3739-7dec-45fc-a1f0-58d7e82806ff-kube-api-access-mhhvk\") pod \"iptables-alerter-gw9bj\" (UID: \"178a3739-7dec-45fc-a1f0-58d7e82806ff\") " pod="openshift-network-operator/iptables-alerter-gw9bj"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706310    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1c5ca752-f1cb-483c-9f16-7b425d081d7b-serviceca\") pod \"node-ca-g55b4\" (UID: \"1c5ca752-f1cb-483c-9f16-7b425d081d7b\") " pod="openshift-image-registry/node-ca-g55b4"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706313    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-sysconfig\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706313    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/247b8f4d-bbc5-44b7-981b-aa42c30c031f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706337    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61608ec6-2071-4478-9bbf-4f8a55085fd9-env-overrides\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-run-multus-certs\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706443    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706468    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-run-netns\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-var-lib-kubelet\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.706810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706521    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-var-lib-kubelet\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-host\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706549    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/247b8f4d-bbc5-44b7-981b-aa42c30c031f-cni-binary-copy\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1618b15-d851-4f6d-a6af-d6e132341463-tmp\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkcp\" (UniqueName: \"kubernetes.io/projected/d928b5bc-8033-4f03-be65-0989dabe8581-kube-api-access-xbkcp\") pod \"node-resolver-6st8d\" (UID: \"d928b5bc-8033-4f03-be65-0989dabe8581\") " pod="openshift-dns/node-resolver-6st8d"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706618    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-slash\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706626    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/178a3739-7dec-45fc-a1f0-58d7e82806ff-iptables-alerter-script\") pod \"iptables-alerter-gw9bj\" (UID: \"178a3739-7dec-45fc-a1f0-58d7e82806ff\") " pod="openshift-network-operator/iptables-alerter-gw9bj"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61608ec6-2071-4478-9bbf-4f8a55085fd9-ovnkube-script-lib\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706626    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-host\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706639    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-var-lib-kubelet\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-os-release\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706708    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vjs\" (UniqueName: \"kubernetes.io/projected/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-kube-api-access-k8vjs\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706736    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1618b15-d851-4f6d-a6af-d6e132341463-etc-tuned\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706768    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1c5ca752-f1cb-483c-9f16-7b425d081d7b-serviceca\") pod \"node-ca-g55b4\" (UID: \"1c5ca752-f1cb-483c-9f16-7b425d081d7b\") " pod="openshift-image-registry/node-ca-g55b4"
Jul 02 21:18:53.707458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/178a3739-7dec-45fc-a1f0-58d7e82806ff-host-slash\") pod \"iptables-alerter-gw9bj\" (UID: \"178a3739-7dec-45fc-a1f0-58d7e82806ff\") " pod="openshift-network-operator/iptables-alerter-gw9bj"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706812    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgvb\" (UniqueName: \"kubernetes.io/projected/1c5ca752-f1cb-483c-9f16-7b425d081d7b-kube-api-access-8kgvb\") pod \"node-ca-g55b4\" (UID: \"1c5ca752-f1cb-483c-9f16-7b425d081d7b\") " pod="openshift-image-registry/node-ca-g55b4"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706840    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7012e89-72a9-4dd4-bc20-10759e43a4dc-konnectivity-ca\") pod \"konnectivity-agent-nm8zf\" (UID: \"e7012e89-72a9-4dd4-bc20-10759e43a4dc\") " pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706851    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/178a3739-7dec-45fc-a1f0-58d7e82806ff-host-slash\") pod \"iptables-alerter-gw9bj\" (UID: \"178a3739-7dec-45fc-a1f0-58d7e82806ff\") " pod="openshift-network-operator/iptables-alerter-gw9bj"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706864    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-run-openvswitch\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706887    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71429679-06c0-480b-8c79-08a2c5682ca1-dbus\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-system-cni-dir\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706938    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-var-lib-openvswitch\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706966    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-cnibin\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706976    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-system-cni-dir\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.706988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-socket-dir-parent\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707031    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-conf-dir\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707057    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-os-release\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707060    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 21:18:53.708181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-sysctl-d\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c5ca752-f1cb-483c-9f16-7b425d081d7b-host\") pod \"node-ca-g55b4\" (UID: \"1c5ca752-f1cb-483c-9f16-7b425d081d7b\") " pod="openshift-image-registry/node-ca-g55b4"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7012e89-72a9-4dd4-bc20-10759e43a4dc-agent-certs\") pod \"konnectivity-agent-nm8zf\" (UID: \"e7012e89-72a9-4dd4-bc20-10759e43a4dc\") " pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707152    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-os-release\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707179    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzt58\" (UniqueName: \"kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58\") pod \"network-check-target-pjcpf\" (UID: \"a723a652-5f75-4d27-acc7-5489c2aba607\") " pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707196    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-sysctl-d\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707202    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61608ec6-2071-4478-9bbf-4f8a55085fd9-ovn-node-metrics-cert\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707210    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c5ca752-f1cb-483c-9f16-7b425d081d7b-host\") pod \"node-ca-g55b4\" (UID: \"1c5ca752-f1cb-483c-9f16-7b425d081d7b\") " pod="openshift-image-registry/node-ca-g55b4"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707267    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c7ccba7-ce5b-45ac-ba36-22db25a36607-cni-binary-copy\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707292    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-daemon-config\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707320    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5cst\" (UniqueName: \"kubernetes.io/projected/30cc5577-2894-4468-9c39-a462c823e44e-kube-api-access-z5cst\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/247b8f4d-bbc5-44b7-981b-aa42c30c031f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707497    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/247b8f4d-bbc5-44b7-981b-aa42c30c031f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.708825 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-sysctl-conf\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707550    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-log-socket\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707569    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-hostroot\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707612    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-etc-kubernetes\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-registration-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-systemd\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-run\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d928b5bc-8033-4f03-be65-0989dabe8581-tmp-dir\") pod \"node-resolver-6st8d\" (UID: \"d928b5bc-8033-4f03-be65-0989dabe8581\") " pod="openshift-dns/node-resolver-6st8d"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707714    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-var-lib-cni-bin\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707769    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-var-lib-cni-multus\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-device-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707813    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-sys-fs\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.709480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707835    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-cni-bin\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707842    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-sysctl-conf\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707858    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61608ec6-2071-4478-9bbf-4f8a55085fd9-ovnkube-config\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707884    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-run-k8s-cni-cncf-io\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707882    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/247b8f4d-bbc5-44b7-981b-aa42c30c031f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707945    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-etc-systemd\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.707975    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.708057    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.708085    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1618b15-d851-4f6d-a6af-d6e132341463-run\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.708089    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-systemd-units\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.708122    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-run-ovn\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.708147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-run-netns\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.708171    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:53.708172    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:53.710139 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:53.708286    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs podName:30cc5577-2894-4468-9c39-a462c823e44e nodeName:}" failed. No retries permitted until 2026-07-02 21:18:54.208269386 +0000 UTC m=+2.941902671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs") pod "network-metrics-daemon-fd5sb" (UID: "30cc5577-2894-4468-9c39-a462c823e44e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:53.711535 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.711509    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1618b15-d851-4f6d-a6af-d6e132341463-etc-tuned\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.712196 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.712175    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1618b15-d851-4f6d-a6af-d6e132341463-tmp\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.716611 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.715975    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgvb\" (UniqueName: \"kubernetes.io/projected/1c5ca752-f1cb-483c-9f16-7b425d081d7b-kube-api-access-8kgvb\") pod \"node-ca-g55b4\" (UID: \"1c5ca752-f1cb-483c-9f16-7b425d081d7b\") " pod="openshift-image-registry/node-ca-g55b4"
Jul 02 21:18:53.717071 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.717048    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhhvk\" (UniqueName: \"kubernetes.io/projected/178a3739-7dec-45fc-a1f0-58d7e82806ff-kube-api-access-mhhvk\") pod \"iptables-alerter-gw9bj\" (UID: \"178a3739-7dec-45fc-a1f0-58d7e82806ff\") " pod="openshift-network-operator/iptables-alerter-gw9bj"
Jul 02 21:18:53.717843 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.717784    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5cst\" (UniqueName: \"kubernetes.io/projected/30cc5577-2894-4468-9c39-a462c823e44e-kube-api-access-z5cst\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:53.718642 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.718620    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhpz\" (UniqueName: \"kubernetes.io/projected/a1618b15-d851-4f6d-a6af-d6e132341463-kube-api-access-8jhpz\") pod \"tuned-4lmlv\" (UID: \"a1618b15-d851-4f6d-a6af-d6e132341463\") " pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.718919 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.718901    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fdjd\" (UniqueName: \"kubernetes.io/projected/247b8f4d-bbc5-44b7-981b-aa42c30c031f-kube-api-access-2fdjd\") pod \"multus-additional-cni-plugins-wdmdj\" (UID: \"247b8f4d-bbc5-44b7-981b-aa42c30c031f\") " pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.808761 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.808725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-cni-bin\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.808761 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.808766    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61608ec6-2071-4478-9bbf-4f8a55085fd9-ovnkube-config\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809021 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.808788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-run-k8s-cni-cncf-io\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.809021 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.808827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.809021 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.808863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-systemd-units\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809021 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.808851    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-cni-bin\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809021 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.808887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-run-ovn\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809021 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.808932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-run-netns\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.809021 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.808936    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-run-k8s-cni-cncf-io\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.809021 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.808938    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-systemd-units\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809021 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.808982    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.809021 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809002    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-run-netns\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-run-systemd\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809056    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-run-ovn\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-node-log\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-cni-dir\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809121    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-run-systemd\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-etc-openvswitch\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809149    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-node-log\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:53.809163    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809171    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-etc-openvswitch\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-kubelet\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-cni-netd\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809242    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-system-cni-dir\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.809577 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:53.809262    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret podName:71429679-06c0-480b-8c79-08a2c5682ca1 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:54.309242568 +0000 UTC m=+3.042875864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret") pod "global-pull-secret-syncer-mdqbk" (UID: "71429679-06c0-480b-8c79-08a2c5682ca1") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809273    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-cni-dir\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zltnq\" (UniqueName: \"kubernetes.io/projected/6c7ccba7-ce5b-45ac-ba36-22db25a36607-kube-api-access-zltnq\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809306    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-system-cni-dir\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d928b5bc-8033-4f03-be65-0989dabe8581-hosts-file\") pod \"node-resolver-6st8d\" (UID: \"d928b5bc-8033-4f03-be65-0989dabe8581\") " pod="openshift-dns/node-resolver-6st8d"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809345    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-cni-netd\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809351    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2mlr\" (UniqueName: \"kubernetes.io/projected/61608ec6-2071-4478-9bbf-4f8a55085fd9-kube-api-access-r2mlr\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809364    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61608ec6-2071-4478-9bbf-4f8a55085fd9-ovnkube-config\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-socket-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71429679-06c0-480b-8c79-08a2c5682ca1-kubelet-config\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809378    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-kubelet\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809422    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d928b5bc-8033-4f03-be65-0989dabe8581-hosts-file\") pod \"node-resolver-6st8d\" (UID: \"d928b5bc-8033-4f03-be65-0989dabe8581\") " pod="openshift-dns/node-resolver-6st8d"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809430    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61608ec6-2071-4478-9bbf-4f8a55085fd9-env-overrides\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-run-multus-certs\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809465    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71429679-06c0-480b-8c79-08a2c5682ca1-kubelet-config\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:53.810220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809476    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809494    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-socket-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809501    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-run-netns\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809513    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-run-multus-certs\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809537    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-run-netns\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-var-lib-kubelet\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809596    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkcp\" (UniqueName: \"kubernetes.io/projected/d928b5bc-8033-4f03-be65-0989dabe8581-kube-api-access-xbkcp\") pod \"node-resolver-6st8d\" (UID: \"d928b5bc-8033-4f03-be65-0989dabe8581\") " pod="openshift-dns/node-resolver-6st8d"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809612    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809625    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-slash\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809646    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-var-lib-kubelet\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61608ec6-2071-4478-9bbf-4f8a55085fd9-ovnkube-script-lib\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-os-release\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809718    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vjs\" (UniqueName: \"kubernetes.io/projected/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-kube-api-access-k8vjs\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809698    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-slash\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7012e89-72a9-4dd4-bc20-10759e43a4dc-konnectivity-ca\") pod \"konnectivity-agent-nm8zf\" (UID: \"e7012e89-72a9-4dd4-bc20-10759e43a4dc\") " pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:18:53.810845 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809795    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-os-release\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809797    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61608ec6-2071-4478-9bbf-4f8a55085fd9-env-overrides\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-run-openvswitch\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71429679-06c0-480b-8c79-08a2c5682ca1-dbus\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809847    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-run-openvswitch\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-var-lib-openvswitch\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-cnibin\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809898    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71429679-06c0-480b-8c79-08a2c5682ca1-dbus\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809921    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-var-lib-openvswitch\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809915    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-socket-dir-parent\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.809947    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-cnibin\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810016    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-socket-dir-parent\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810034    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-conf-dir\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7012e89-72a9-4dd4-bc20-10759e43a4dc-agent-certs\") pod \"konnectivity-agent-nm8zf\" (UID: \"e7012e89-72a9-4dd4-bc20-10759e43a4dc\") " pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810091    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzt58\" (UniqueName: \"kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58\") pod \"network-check-target-pjcpf\" (UID: \"a723a652-5f75-4d27-acc7-5489c2aba607\") " pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:18:53.811561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810103    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-conf-dir\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810120    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61608ec6-2071-4478-9bbf-4f8a55085fd9-ovn-node-metrics-cert\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c7ccba7-ce5b-45ac-ba36-22db25a36607-cni-binary-copy\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810282    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61608ec6-2071-4478-9bbf-4f8a55085fd9-ovnkube-script-lib\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810307    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-daemon-config\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810323    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7012e89-72a9-4dd4-bc20-10759e43a4dc-konnectivity-ca\") pod \"konnectivity-agent-nm8zf\" (UID: \"e7012e89-72a9-4dd4-bc20-10759e43a4dc\") " pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-log-socket\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810410    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-hostroot\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810446    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810486    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-etc-kubernetes\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810488    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-log-socket\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810497    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-hostroot\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-registration-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810547    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810565    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-etc-kubernetes\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812217 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d928b5bc-8033-4f03-be65-0989dabe8581-tmp-dir\") pod \"node-resolver-6st8d\" (UID: \"d928b5bc-8033-4f03-be65-0989dabe8581\") " pod="openshift-dns/node-resolver-6st8d"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810594    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.810612    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-registration-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.811968    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6c7ccba7-ce5b-45ac-ba36-22db25a36607-multus-daemon-config\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.812099    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61608ec6-2071-4478-9bbf-4f8a55085fd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.812101    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-var-lib-cni-bin\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.812127    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c7ccba7-ce5b-45ac-ba36-22db25a36607-cni-binary-copy\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.812175    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-var-lib-cni-multus\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.812182    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-var-lib-cni-bin\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.812260    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6c7ccba7-ce5b-45ac-ba36-22db25a36607-host-var-lib-cni-multus\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.812280    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-device-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.812313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-sys-fs\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.812346    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-device-dir\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.812418    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-sys-fs\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.812883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.812801    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61608ec6-2071-4478-9bbf-4f8a55085fd9-ovn-node-metrics-cert\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.813582 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.813559    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7012e89-72a9-4dd4-bc20-10759e43a4dc-agent-certs\") pod \"konnectivity-agent-nm8zf\" (UID: \"e7012e89-72a9-4dd4-bc20-10759e43a4dc\") " pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:18:53.815065 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.814757    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d928b5bc-8033-4f03-be65-0989dabe8581-tmp-dir\") pod \"node-resolver-6st8d\" (UID: \"d928b5bc-8033-4f03-be65-0989dabe8581\") " pod="openshift-dns/node-resolver-6st8d"
Jul 02 21:18:53.821172 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:53.821092    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:53.821172 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:53.821121    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:53.821172 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:53.821137    2564 projected.go:196] Error preparing data for projected volume kube-api-access-lzt58 for pod openshift-network-diagnostics/network-check-target-pjcpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:53.821396 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:53.821228    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58 podName:a723a652-5f75-4d27-acc7-5489c2aba607 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:54.321210426 +0000 UTC m=+3.054843735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lzt58" (UniqueName: "kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58") pod "network-check-target-pjcpf" (UID: "a723a652-5f75-4d27-acc7-5489c2aba607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:53.822447 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.822425    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2mlr\" (UniqueName: \"kubernetes.io/projected/61608ec6-2071-4478-9bbf-4f8a55085fd9-kube-api-access-r2mlr\") pod \"ovnkube-node-cd8lj\" (UID: \"61608ec6-2071-4478-9bbf-4f8a55085fd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.822672 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.822650    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zltnq\" (UniqueName: \"kubernetes.io/projected/6c7ccba7-ce5b-45ac-ba36-22db25a36607-kube-api-access-zltnq\") pod \"multus-bllj5\" (UID: \"6c7ccba7-ce5b-45ac-ba36-22db25a36607\") " pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.823681 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.823662    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vjs\" (UniqueName: \"kubernetes.io/projected/5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3-kube-api-access-k8vjs\") pod \"aws-ebs-csi-driver-node-hctd7\" (UID: \"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.823764 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.823691    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkcp\" (UniqueName: \"kubernetes.io/projected/d928b5bc-8033-4f03-be65-0989dabe8581-kube-api-access-xbkcp\") pod \"node-resolver-6st8d\" (UID: \"d928b5bc-8033-4f03-be65-0989dabe8581\") " pod="openshift-dns/node-resolver-6st8d"
Jul 02 21:18:53.832790 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.832745    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.ec2.internal" event={"ID":"fddbb2f027add0a069d4010707bb8cdd","Type":"ContainerStarted","Data":"aedf9ee2af6bb31b382c678a48358beb42088542d766136b52b38fc1f265182c"}
Jul 02 21:18:53.833868 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.833849    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal" event={"ID":"e3a57b42eebfe2a79b8f7b7b30c86f49","Type":"ContainerStarted","Data":"a706621ec1805701ee12db3010c6857d4e464d887fc34ddf75820306aad95e88"}
Jul 02 21:18:53.897097 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.897051    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4lmlv"
Jul 02 21:18:53.910321 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.910295    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gw9bj"
Jul 02 21:18:53.919246 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.919222    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g55b4"
Jul 02 21:18:53.927978 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.927958    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wdmdj"
Jul 02 21:18:53.935682 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.935659    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bllj5"
Jul 02 21:18:53.944972 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.944952    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7"
Jul 02 21:18:53.953692 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.953666    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:18:53.961272 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.961250    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6st8d"
Jul 02 21:18:53.967839 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:53.967815    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:18:54.214672 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.214581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:54.214838 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:54.214732    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:54.214838 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:54.214802    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs podName:30cc5577-2894-4468-9c39-a462c823e44e nodeName:}" failed. No retries permitted until 2026-07-02 21:18:55.214781989 +0000 UTC m=+3.948415291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs") pod "network-metrics-daemon-fd5sb" (UID: "30cc5577-2894-4468-9c39-a462c823e44e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:54.315127 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.315070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:54.315311 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:54.315238    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:54.315311 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:54.315305    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret podName:71429679-06c0-480b-8c79-08a2c5682ca1 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:55.315286865 +0000 UTC m=+4.048920168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret") pod "global-pull-secret-syncer-mdqbk" (UID: "71429679-06c0-480b-8c79-08a2c5682ca1") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:54.415874 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.415829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzt58\" (UniqueName: \"kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58\") pod \"network-check-target-pjcpf\" (UID: \"a723a652-5f75-4d27-acc7-5489c2aba607\") " pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:18:54.416072 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:54.416051    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:54.416141 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:54.416082    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:54.416141 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:54.416095    2564 projected.go:196] Error preparing data for projected volume kube-api-access-lzt58 for pod openshift-network-diagnostics/network-check-target-pjcpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:54.416230 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:54.416160    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58 podName:a723a652-5f75-4d27-acc7-5489c2aba607 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:55.416146469 +0000 UTC m=+4.149779758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzt58" (UniqueName: "kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58") pod "network-check-target-pjcpf" (UID: "a723a652-5f75-4d27-acc7-5489c2aba607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:54.450181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.450156    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:54.646349 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.646241    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 21:13:53 +0000 UTC" deadline="2028-04-03 23:03:26.050622236 +0000 UTC"
Jul 02 21:18:54.646349 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.646303    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15385h44m31.40432347s"
Jul 02 21:18:54.711894 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:54.711865    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247b8f4d_bbc5_44b7_981b_aa42c30c031f.slice/crio-551e3ce17fa85b578c300817187aeff824046ef64645a031faefdd74441632a1 WatchSource:0}: Error finding container 551e3ce17fa85b578c300817187aeff824046ef64645a031faefdd74441632a1: Status 404 returned error can't find the container with id 551e3ce17fa85b578c300817187aeff824046ef64645a031faefdd74441632a1
Jul 02 21:18:54.712430 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:54.712379    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7012e89_72a9_4dd4_bc20_10759e43a4dc.slice/crio-b9f6770c63d8aa3267bdddb15b181de51225f07ca27b4d53183ad7ac608adbb3 WatchSource:0}: Error finding container b9f6770c63d8aa3267bdddb15b181de51225f07ca27b4d53183ad7ac608adbb3: Status 404 returned error can't find the container with id b9f6770c63d8aa3267bdddb15b181de51225f07ca27b4d53183ad7ac608adbb3
Jul 02 21:18:54.737265 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:54.737240    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61608ec6_2071_4478_9bbf_4f8a55085fd9.slice/crio-b1fbf7c51d7c3134cf23b7d28852eb413b872b2878a96f316e0d46264b0acb5c WatchSource:0}: Error finding container b1fbf7c51d7c3134cf23b7d28852eb413b872b2878a96f316e0d46264b0acb5c: Status 404 returned error can't find the container with id b1fbf7c51d7c3134cf23b7d28852eb413b872b2878a96f316e0d46264b0acb5c
Jul 02 21:18:54.738430 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:54.738409    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd928b5bc_8033_4f03_be65_0989dabe8581.slice/crio-541c172c87c35d79d7995f6c7cca10a19e9ce687676327805ed1b05b712ef2ab WatchSource:0}: Error finding container 541c172c87c35d79d7995f6c7cca10a19e9ce687676327805ed1b05b712ef2ab: Status 404 returned error can't find the container with id 541c172c87c35d79d7995f6c7cca10a19e9ce687676327805ed1b05b712ef2ab
Jul 02 21:18:54.739588 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:54.739559    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178a3739_7dec_45fc_a1f0_58d7e82806ff.slice/crio-7a8e50e56b2f46a607c1d394d8e81ca57422db2c4a8b34f1ff43748e6c93f6da WatchSource:0}: Error finding container 7a8e50e56b2f46a607c1d394d8e81ca57422db2c4a8b34f1ff43748e6c93f6da: Status 404 returned error can't find the container with id 7a8e50e56b2f46a607c1d394d8e81ca57422db2c4a8b34f1ff43748e6c93f6da
Jul 02 21:18:54.740768 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:54.740745    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1618b15_d851_4f6d_a6af_d6e132341463.slice/crio-2c19fe266b7e91c8275e17683ef724816476b157130758975df5b8460db97d0d WatchSource:0}: Error finding container 2c19fe266b7e91c8275e17683ef724816476b157130758975df5b8460db97d0d: Status 404 returned error can't find the container with id 2c19fe266b7e91c8275e17683ef724816476b157130758975df5b8460db97d0d
Jul 02 21:18:54.742371 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:54.742151    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7ccba7_ce5b_45ac_ba36_22db25a36607.slice/crio-78f90590a254441cffd09591e4d4904521261484b3a53d2538597c8a4832d049 WatchSource:0}: Error finding container 78f90590a254441cffd09591e4d4904521261484b3a53d2538597c8a4832d049: Status 404 returned error can't find the container with id 78f90590a254441cffd09591e4d4904521261484b3a53d2538597c8a4832d049
Jul 02 21:18:54.743600 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:54.743487    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1f1ff8_c43a_4966_8a3c_4022f7fe05f3.slice/crio-804ef81c3639e1fa7dd133052dc4e9fae1be65037111ea43ac11ab13561d1a44 WatchSource:0}: Error finding container 804ef81c3639e1fa7dd133052dc4e9fae1be65037111ea43ac11ab13561d1a44: Status 404 returned error can't find the container with id 804ef81c3639e1fa7dd133052dc4e9fae1be65037111ea43ac11ab13561d1a44
Jul 02 21:18:54.745508 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:18:54.745285    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5ca752_f1cb_483c_9f16_7b425d081d7b.slice/crio-c72b5aa8e976deac91af25750c8bc0aa1f4b21f24348cacd9ad32c6bb3919d13 WatchSource:0}: Error finding container c72b5aa8e976deac91af25750c8bc0aa1f4b21f24348cacd9ad32c6bb3919d13: Status 404 returned error can't find the container with id c72b5aa8e976deac91af25750c8bc0aa1f4b21f24348cacd9ad32c6bb3919d13
Jul 02 21:18:54.828944 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.828767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:18:54.829064 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.828797    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:54.829109 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:54.829063    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:18:54.829148 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:54.829106    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:18:54.839906 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.839878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4lmlv" event={"ID":"a1618b15-d851-4f6d-a6af-d6e132341463","Type":"ContainerStarted","Data":"2c19fe266b7e91c8275e17683ef724816476b157130758975df5b8460db97d0d"}
Jul 02 21:18:54.842574 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.842546    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gw9bj" event={"ID":"178a3739-7dec-45fc-a1f0-58d7e82806ff","Type":"ContainerStarted","Data":"7a8e50e56b2f46a607c1d394d8e81ca57422db2c4a8b34f1ff43748e6c93f6da"}
Jul 02 21:18:54.843799 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.843779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6st8d" event={"ID":"d928b5bc-8033-4f03-be65-0989dabe8581","Type":"ContainerStarted","Data":"541c172c87c35d79d7995f6c7cca10a19e9ce687676327805ed1b05b712ef2ab"}
Jul 02 21:18:54.847184 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.847161    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj" event={"ID":"61608ec6-2071-4478-9bbf-4f8a55085fd9","Type":"ContainerStarted","Data":"b1fbf7c51d7c3134cf23b7d28852eb413b872b2878a96f316e0d46264b0acb5c"}
Jul 02 21:18:54.848265 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.848242    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdmdj" event={"ID":"247b8f4d-bbc5-44b7-981b-aa42c30c031f","Type":"ContainerStarted","Data":"551e3ce17fa85b578c300817187aeff824046ef64645a031faefdd74441632a1"}
Jul 02 21:18:54.850101 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.850080    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.ec2.internal" event={"ID":"fddbb2f027add0a069d4010707bb8cdd","Type":"ContainerStarted","Data":"912b613c5bea4c97ebaf775b1713c532974fe8ecf202c8061c69d974a1eda959"}
Jul 02 21:18:54.852221 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.852199    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bllj5" event={"ID":"6c7ccba7-ce5b-45ac-ba36-22db25a36607","Type":"ContainerStarted","Data":"78f90590a254441cffd09591e4d4904521261484b3a53d2538597c8a4832d049"}
Jul 02 21:18:54.853869 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.853849    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nm8zf" event={"ID":"e7012e89-72a9-4dd4-bc20-10759e43a4dc","Type":"ContainerStarted","Data":"b9f6770c63d8aa3267bdddb15b181de51225f07ca27b4d53183ad7ac608adbb3"}
Jul 02 21:18:54.856257 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.856237    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g55b4" event={"ID":"1c5ca752-f1cb-483c-9f16-7b425d081d7b","Type":"ContainerStarted","Data":"c72b5aa8e976deac91af25750c8bc0aa1f4b21f24348cacd9ad32c6bb3919d13"}
Jul 02 21:18:54.857632 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.857611    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7" event={"ID":"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3","Type":"ContainerStarted","Data":"804ef81c3639e1fa7dd133052dc4e9fae1be65037111ea43ac11ab13561d1a44"}
Jul 02 21:18:54.864162 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:54.864121    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-121.ec2.internal" podStartSLOduration=2.864108672 podStartE2EDuration="2.864108672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:18:54.863883155 +0000 UTC m=+3.597516464" watchObservedRunningTime="2026-07-02 21:18:54.864108672 +0000 UTC m=+3.597741980"
Jul 02 21:18:55.005623 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:55.005599    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 21:18:55.222695 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:55.222604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:55.222855 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:55.222777    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:55.222919 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:55.222856    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs podName:30cc5577-2894-4468-9c39-a462c823e44e nodeName:}" failed. No retries permitted until 2026-07-02 21:18:57.222835257 +0000 UTC m=+5.956468557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs") pod "network-metrics-daemon-fd5sb" (UID: "30cc5577-2894-4468-9c39-a462c823e44e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:55.323145 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:55.323106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:55.323314 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:55.323288    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:55.323384 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:55.323358    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret podName:71429679-06c0-480b-8c79-08a2c5682ca1 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:57.323340006 +0000 UTC m=+6.056973294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret") pod "global-pull-secret-syncer-mdqbk" (UID: "71429679-06c0-480b-8c79-08a2c5682ca1") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:55.423692 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:55.423644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzt58\" (UniqueName: \"kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58\") pod \"network-check-target-pjcpf\" (UID: \"a723a652-5f75-4d27-acc7-5489c2aba607\") " pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:18:55.423888 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:55.423860    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:55.423888 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:55.423878    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:55.424057 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:55.423890    2564 projected.go:196] Error preparing data for projected volume kube-api-access-lzt58 for pod openshift-network-diagnostics/network-check-target-pjcpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:55.424057 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:55.423951    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58 podName:a723a652-5f75-4d27-acc7-5489c2aba607 nodeName:}" failed. No retries permitted until 2026-07-02 21:18:57.423933002 +0000 UTC m=+6.157566305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzt58" (UniqueName: "kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58") pod "network-check-target-pjcpf" (UID: "a723a652-5f75-4d27-acc7-5489c2aba607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:55.647252 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:55.647127    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 21:13:53 +0000 UTC" deadline="2028-02-07 08:24:47.113723538 +0000 UTC"
Jul 02 21:18:55.647252 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:55.647192    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14027h5m51.46653628s"
Jul 02 21:18:55.828776 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:55.828747    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:55.828933 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:55.828894    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:18:55.885369 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:55.885325    2564 generic.go:356] "Generic (PLEG): container finished" podID="e3a57b42eebfe2a79b8f7b7b30c86f49" containerID="fc706479a942904e832e10000a3c1dc91f4b806f1e6dae5c0fe3c9ea09e68b87" exitCode=0
Jul 02 21:18:55.886500 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:55.886472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal" event={"ID":"e3a57b42eebfe2a79b8f7b7b30c86f49","Type":"ContainerDied","Data":"fc706479a942904e832e10000a3c1dc91f4b806f1e6dae5c0fe3c9ea09e68b87"}
Jul 02 21:18:56.828890 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:56.828292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:56.828890 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:56.828339    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:18:56.828890 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:56.828426    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:18:56.828890 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:56.828510    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:18:56.912436 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:56.911756    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal" event={"ID":"e3a57b42eebfe2a79b8f7b7b30c86f49","Type":"ContainerStarted","Data":"e43b4f58513f388fc9d25c034e7f4fd5038f927569068e42579759b25efd91ea"}
Jul 02 21:18:57.239897 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:57.239178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:57.239897 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:57.239393    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:57.239897 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:57.239458    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs podName:30cc5577-2894-4468-9c39-a462c823e44e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:01.239438583 +0000 UTC m=+9.973071872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs") pod "network-metrics-daemon-fd5sb" (UID: "30cc5577-2894-4468-9c39-a462c823e44e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:18:57.340379 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:57.339783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:57.340379 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:57.339939    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:57.340379 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:57.340015    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret podName:71429679-06c0-480b-8c79-08a2c5682ca1 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:01.339979798 +0000 UTC m=+10.073613107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret") pod "global-pull-secret-syncer-mdqbk" (UID: "71429679-06c0-480b-8c79-08a2c5682ca1") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:18:57.441287 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:57.440638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzt58\" (UniqueName: \"kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58\") pod \"network-check-target-pjcpf\" (UID: \"a723a652-5f75-4d27-acc7-5489c2aba607\") " pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:18:57.441287 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:57.440808    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:18:57.441287 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:57.440832    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:18:57.441287 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:57.440845    2564 projected.go:196] Error preparing data for projected volume kube-api-access-lzt58 for pod openshift-network-diagnostics/network-check-target-pjcpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:57.441287 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:57.440899    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58 podName:a723a652-5f75-4d27-acc7-5489c2aba607 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:01.440882105 +0000 UTC m=+10.174515415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzt58" (UniqueName: "kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58") pod "network-check-target-pjcpf" (UID: "a723a652-5f75-4d27-acc7-5489c2aba607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:18:57.829058 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:57.828334    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:57.829520 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:57.829182    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:18:58.829345 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:58.829312    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:18:58.829852 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:58.829448    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:18:58.829852 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:58.829813    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:18:58.829982 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:58.829896    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:18:59.828869 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:18:59.828752    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:18:59.829059 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:18:59.829031    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:00.829081 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:00.829006    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:00.829542 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:00.829138    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:00.829542 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:00.829493    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:00.829653 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:00.829595    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:01.275060 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:01.274961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:01.275220 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:01.275129    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:01.275220 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:01.275213    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs podName:30cc5577-2894-4468-9c39-a462c823e44e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:09.275191279 +0000 UTC m=+18.008824578 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs") pod "network-metrics-daemon-fd5sb" (UID: "30cc5577-2894-4468-9c39-a462c823e44e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:01.376293 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:01.376251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:01.376482 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:01.376409    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:01.376557 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:01.376488    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret podName:71429679-06c0-480b-8c79-08a2c5682ca1 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:09.376468812 +0000 UTC m=+18.110102098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret") pod "global-pull-secret-syncer-mdqbk" (UID: "71429679-06c0-480b-8c79-08a2c5682ca1") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:01.477187 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:01.477149    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzt58\" (UniqueName: \"kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58\") pod \"network-check-target-pjcpf\" (UID: \"a723a652-5f75-4d27-acc7-5489c2aba607\") " pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:01.477326 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:01.477307    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:01.477374 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:01.477326    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:01.477374 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:01.477339    2564 projected.go:196] Error preparing data for projected volume kube-api-access-lzt58 for pod openshift-network-diagnostics/network-check-target-pjcpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:01.477438 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:01.477401    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58 podName:a723a652-5f75-4d27-acc7-5489c2aba607 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:09.477383672 +0000 UTC m=+18.211016958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzt58" (UniqueName: "kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58") pod "network-check-target-pjcpf" (UID: "a723a652-5f75-4d27-acc7-5489c2aba607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:01.830075 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:01.829817    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:01.830075 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:01.829942    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:02.828923 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:02.828888    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:02.829096 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:02.829045    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:02.829156 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:02.829095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:02.829248 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:02.829202    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:03.828613 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:03.828576    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:03.829082 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:03.828715    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:04.828693 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:04.828655    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:04.829187 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:04.828700    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:04.829187 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:04.828803    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:04.829187 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:04.828924    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:05.828395 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:05.828362    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:05.828578 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:05.828477    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:06.828289 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:06.828255    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:06.828289 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:06.828266    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:06.828762 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:06.828395    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:06.828762 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:06.828522    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:07.828553 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:07.828520    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:07.828978 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:07.828648    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:08.829289 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:08.829260    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:08.829289 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:08.829285    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:08.829733 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:08.829360    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:08.829733 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:08.829474    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:09.336658 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:09.336623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:09.336840 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:09.336788    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:09.336915 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:09.336847    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs podName:30cc5577-2894-4468-9c39-a462c823e44e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:25.336831355 +0000 UTC m=+34.070464641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs") pod "network-metrics-daemon-fd5sb" (UID: "30cc5577-2894-4468-9c39-a462c823e44e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:09.437370 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:09.437336    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:09.437535 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:09.437453    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:09.437535 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:09.437515    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret podName:71429679-06c0-480b-8c79-08a2c5682ca1 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:25.437498143 +0000 UTC m=+34.171131432 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret") pod "global-pull-secret-syncer-mdqbk" (UID: "71429679-06c0-480b-8c79-08a2c5682ca1") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:09.538415 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:09.538376    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzt58\" (UniqueName: \"kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58\") pod \"network-check-target-pjcpf\" (UID: \"a723a652-5f75-4d27-acc7-5489c2aba607\") " pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:09.538590 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:09.538515    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:09.538590 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:09.538532    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:09.538590 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:09.538541    2564 projected.go:196] Error preparing data for projected volume kube-api-access-lzt58 for pod openshift-network-diagnostics/network-check-target-pjcpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:09.538590 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:09.538586    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58 podName:a723a652-5f75-4d27-acc7-5489c2aba607 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:25.538574702 +0000 UTC m=+34.272207988 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzt58" (UniqueName: "kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58") pod "network-check-target-pjcpf" (UID: "a723a652-5f75-4d27-acc7-5489c2aba607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:09.829094 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:09.829012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:09.829232 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:09.829175    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:10.828862 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:10.828824    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:10.828862 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:10.828857    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:10.829378 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:10.828957    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:10.829378 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:10.829074    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:11.829245 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:11.829212    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:11.829676 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:11.829326    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:12.828594 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:12.828555    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:12.828765 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:12.828556    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:12.828765 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:12.828678    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:12.828864 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:12.828783    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:13.829336 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.829172    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:13.829879 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:13.829427    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:13.949762 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.949731    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g55b4" event={"ID":"1c5ca752-f1cb-483c-9f16-7b425d081d7b","Type":"ContainerStarted","Data":"cd60a25dbe3d089a726a36adac2008fee1e141b6aaec4bfe07fbd3c6a4815e7b"}
Jul 02 21:19:13.951090 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.951067    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7" event={"ID":"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3","Type":"ContainerStarted","Data":"0b51759515a428fe13dfde8ae4e4b42f226310ea58c4b115bcbe359586853c6f"}
Jul 02 21:19:13.952336 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.952316    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4lmlv" event={"ID":"a1618b15-d851-4f6d-a6af-d6e132341463","Type":"ContainerStarted","Data":"3f9fc5f6d25d993a274f5f5f2ed4344bfb2ae644511804185681a15b79a15754"}
Jul 02 21:19:13.953479 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.953457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6st8d" event={"ID":"d928b5bc-8033-4f03-be65-0989dabe8581","Type":"ContainerStarted","Data":"322e6748717967613111b50a509913bd6421d622d41616f077cc30667af8ed37"}
Jul 02 21:19:13.955639 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.955622    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj" event={"ID":"61608ec6-2071-4478-9bbf-4f8a55085fd9","Type":"ContainerStarted","Data":"363753cfbc8db16a60308ec1e451fee0e711dd47a55f63490a0e553df27be602"}
Jul 02 21:19:13.955724 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.955669    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj" event={"ID":"61608ec6-2071-4478-9bbf-4f8a55085fd9","Type":"ContainerStarted","Data":"d301a9b6e6239d90e4ee5bd0fd54409bc46b67258ade3cb5d92de51f8d5fd382"}
Jul 02 21:19:13.955724 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.955699    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj" event={"ID":"61608ec6-2071-4478-9bbf-4f8a55085fd9","Type":"ContainerStarted","Data":"72d39b33f10e464558dd17b92e13faeaf566008ab245f8ed5562ec45462bd44e"}
Jul 02 21:19:13.955802 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.955727    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj" event={"ID":"61608ec6-2071-4478-9bbf-4f8a55085fd9","Type":"ContainerStarted","Data":"973445c2d6bdb8dd1b4b774a38d9cac4690eee9416f98a6d739c6ef3b6c822c2"}
Jul 02 21:19:13.957001 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.956968    2564 generic.go:356] "Generic (PLEG): container finished" podID="247b8f4d-bbc5-44b7-981b-aa42c30c031f" containerID="06997f4b816bb22de0f06ec5dac4001922516dd25e568a3e61450d78bccbd5f4" exitCode=0
Jul 02 21:19:13.957072 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.957050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdmdj" event={"ID":"247b8f4d-bbc5-44b7-981b-aa42c30c031f","Type":"ContainerDied","Data":"06997f4b816bb22de0f06ec5dac4001922516dd25e568a3e61450d78bccbd5f4"}
Jul 02 21:19:13.958386 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.958318    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bllj5" event={"ID":"6c7ccba7-ce5b-45ac-ba36-22db25a36607","Type":"ContainerStarted","Data":"f25eae9f882fcd4f12a8a922ee572b5cbaee33eaeffaa6289cd453ac0eb3d14f"}
Jul 02 21:19:13.959625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.959536    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nm8zf" event={"ID":"e7012e89-72a9-4dd4-bc20-10759e43a4dc","Type":"ContainerStarted","Data":"5718018b8a0f1e062a9b49d3cf2abbd2e38eb690a299da03959b2592a20336c1"}
Jul 02 21:19:13.971954 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:13.971916    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-121.ec2.internal" podStartSLOduration=20.971904532 podStartE2EDuration="20.971904532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:18:56.930220571 +0000 UTC m=+5.663853899" watchObservedRunningTime="2026-07-02 21:19:13.971904532 +0000 UTC m=+22.705537840"
Jul 02 21:19:14.002704 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.002664    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4lmlv" podStartSLOduration=4.42207575 podStartE2EDuration="23.00265109s" totalImagesPullingTime="18.58057534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:14.002535588 +0000 UTC m=+22.736168908" watchObservedRunningTime="2026-07-02 21:19:14.00265109 +0000 UTC m=+22.736284399"
Jul 02 21:19:14.003088 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.003055    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g55b4" podStartSLOduration=4.430649861 podStartE2EDuration="23.003045708s" totalImagesPullingTime="18.572395847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:13.97259606 +0000 UTC m=+22.706229370" watchObservedRunningTime="2026-07-02 21:19:14.003045708 +0000 UTC m=+22.736679077"
Jul 02 21:19:14.071937 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.071700    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6st8d" podStartSLOduration=3.489350734 podStartE2EDuration="22.071680843s" totalImagesPullingTime="18.582330109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:14.037365899 +0000 UTC m=+22.770999209" watchObservedRunningTime="2026-07-02 21:19:14.071680843 +0000 UTC m=+22.805314152"
Jul 02 21:19:14.110381 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.110323    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nm8zf" podStartSLOduration=7.974077004 podStartE2EDuration="22.110301732s" totalImagesPullingTime="14.136224728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:14.071526586 +0000 UTC m=+22.805159895" watchObservedRunningTime="2026-07-02 21:19:14.110301732 +0000 UTC m=+22.843935041"
Jul 02 21:19:14.136248 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.136196    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bllj5" podStartSLOduration=4.500075387 podStartE2EDuration="23.136179671s" totalImagesPullingTime="18.636104284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:14.136050054 +0000 UTC m=+22.869683387" watchObservedRunningTime="2026-07-02 21:19:14.136179671 +0000 UTC m=+22.869812980"
Jul 02 21:19:14.313958 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.313921    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:14.444519 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.444438    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:14.790831 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.790801    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 21:19:14.828871 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.828847    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:14.828871 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.828859    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:14.829086 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:14.828963    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:14.829144 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:14.829081    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:14.964027 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.963976    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7" event={"ID":"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3","Type":"ContainerStarted","Data":"8d77a55e7065b6043dafc9375ea4cb0237dd085491963dad93f6a1668d2abbd1"}
Jul 02 21:19:14.965602 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.965568    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gw9bj" event={"ID":"178a3739-7dec-45fc-a1f0-58d7e82806ff","Type":"ContainerStarted","Data":"46a2a1436745f76135345f36ec2bbe61bb7be5607657f3098a2bad556e5443a1"}
Jul 02 21:19:14.970548 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.970516    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj" event={"ID":"61608ec6-2071-4478-9bbf-4f8a55085fd9","Type":"ContainerStarted","Data":"29f9badaa1253d9bb7192daaa166cb95538271247669fe3ff5e9ec54d089f0bd"}
Jul 02 21:19:14.970678 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.970604    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj" event={"ID":"61608ec6-2071-4478-9bbf-4f8a55085fd9","Type":"ContainerStarted","Data":"43890cde8f8040aead9551f4e8c37a8c6569845559a4d14784a9b0c4814bb835"}
Jul 02 21:19:14.987811 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:14.987720    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gw9bj" podStartSLOduration=5.406863029 podStartE2EDuration="23.987704524s" totalImagesPullingTime="18.580841495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:14.987251348 +0000 UTC m=+23.720884658" watchObservedRunningTime="2026-07-02 21:19:14.987704524 +0000 UTC m=+23.721337834"
Jul 02 21:19:15.140158 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.140121    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:15.239345 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.239218    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:15.690952 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.690806    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T21:19:14.790827302Z","UUID":"c49b58e2-9624-421b-b951-1878eaf23a1b","Handler":null,"Name":"","Endpoint":""}
Jul 02 21:19:15.692372 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.692351    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 21:19:15.692536 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.692388    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 21:19:15.742694 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.742666    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:19:15.743264 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.743245    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:19:15.828809 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.828781    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:15.829011 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:15.828933    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:15.975479 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.975390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7" event={"ID":"5e1f1ff8-c43a-4966-8a3c-4022f7fe05f3","Type":"ContainerStarted","Data":"faa723a21076f003bfed2e459f24ce92f2a2be02d0583048fb922f6620fd9995"}
Jul 02 21:19:15.975965 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.975917    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:19:15.976438 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.976416    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nm8zf"
Jul 02 21:19:15.995012 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:15.994942    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hctd7" podStartSLOduration=4.294101867 podStartE2EDuration="24.994924099s" totalImagesPullingTime="20.700822232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:15.994627001 +0000 UTC m=+24.728260327" watchObservedRunningTime="2026-07-02 21:19:15.994924099 +0000 UTC m=+24.728557409"
Jul 02 21:19:16.072290 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:16.072242    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:16.828821 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:16.828628    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:16.829098 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:16.828628    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:16.829098 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:16.828913    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:16.829098 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:16.828956    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:16.981769 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:16.981726    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj" event={"ID":"61608ec6-2071-4478-9bbf-4f8a55085fd9","Type":"ContainerStarted","Data":"83180248ecf74ee9330db31c06689460a12b652a14b5b7ea9d1605b53abc5523"}
Jul 02 21:19:17.828891 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:17.828862    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:17.829112 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:17.829006    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:18.734575 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:18.734306    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:18.828319 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:18.828292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:18.828498 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:18.828292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:18.828498 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:18.828392    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:18.828604 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:18.828490    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:18.834220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:18.834194    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:18.989523 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:18.989442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj" event={"ID":"61608ec6-2071-4478-9bbf-4f8a55085fd9","Type":"ContainerStarted","Data":"dce83db4053a9fc54420171481c44429f583363a39755d08fbe4b632b1c74a04"}
Jul 02 21:19:18.989678 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:18.989635    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:19:18.989744 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:18.989690    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:19:18.991339 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:18.991314    2564 generic.go:356] "Generic (PLEG): container finished" podID="247b8f4d-bbc5-44b7-981b-aa42c30c031f" containerID="5e90547373b5204c1a5272be9c2bdc81f4cb847c319addb0af0c924f950ec4e5" exitCode=0
Jul 02 21:19:18.991443 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:18.991355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdmdj" event={"ID":"247b8f4d-bbc5-44b7-981b-aa42c30c031f","Type":"ContainerDied","Data":"5e90547373b5204c1a5272be9c2bdc81f4cb847c319addb0af0c924f950ec4e5"}
Jul 02 21:19:19.006302 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:19.006281    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:19:19.023912 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:19.023868    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj" podStartSLOduration=8.373051056 podStartE2EDuration="27.023854422s" totalImagesPullingTime="18.650803366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:19.022063955 +0000 UTC m=+27.755697276" watchObservedRunningTime="2026-07-02 21:19:19.023854422 +0000 UTC m=+27.757487730"
Jul 02 21:19:19.829477 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:19.829408    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:19.829912 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:19.829554    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:19.994314 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:19.994279    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:19:20.008734 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:20.008709    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:19:20.307628 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:20.307599    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mdqbk"]
Jul 02 21:19:20.307778 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:20.307697    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:20.307844 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:20.307774    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:20.314879 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:20.314854    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fd5sb"]
Jul 02 21:19:20.315017 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:20.314951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:20.315102 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:20.315080    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:20.315561 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:20.315543    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pjcpf"]
Jul 02 21:19:20.315641 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:20.315630    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:20.315725 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:20.315705    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:20.997418 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:20.997387    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:20.998983 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:20.998959    2564 generic.go:356] "Generic (PLEG): container finished" podID="247b8f4d-bbc5-44b7-981b-aa42c30c031f" containerID="363b413a8023330cc81c9e0b24386ee146fcc85d3904a4742a25f013e92b248b" exitCode=0
Jul 02 21:19:20.999107 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:20.999024    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdmdj" event={"ID":"247b8f4d-bbc5-44b7-981b-aa42c30c031f","Type":"ContainerDied","Data":"363b413a8023330cc81c9e0b24386ee146fcc85d3904a4742a25f013e92b248b"}
Jul 02 21:19:21.057922 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:21.057894    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:21.831253 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:21.831232    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:21.831428 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:21.831233    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:21.831428 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:21.831233    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:21.831428 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:21.831410    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:21.831533 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:21.831465    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:21.831533 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:21.831317    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:21.934314 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:21.934277    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:22.034235 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:22.034196    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:23.006404 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:23.006373    2564 generic.go:356] "Generic (PLEG): container finished" podID="247b8f4d-bbc5-44b7-981b-aa42c30c031f" containerID="2b741cc3735a27d1c0ff85e0fcffc194e59776ab1f3c11be03d8456b4850276c" exitCode=0
Jul 02 21:19:23.006573 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:23.006434    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdmdj" event={"ID":"247b8f4d-bbc5-44b7-981b-aa42c30c031f","Type":"ContainerDied","Data":"2b741cc3735a27d1c0ff85e0fcffc194e59776ab1f3c11be03d8456b4850276c"}
Jul 02 21:19:23.364875 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:23.364841    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:23.461117 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:23.461082    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:23.829162 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:23.829129    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:23.829162 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:23.829157    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:23.829401 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:23.829129    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:23.829401 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:23.829256    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:23.829401 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:23.829356    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:23.829544 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:23.829456    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:24.070275 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:24.070218    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:24.178383 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:24.178292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:24.271724 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:24.271688    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:24.381111 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:24.381071    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:24.476565 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:24.476509    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:24.567190 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:24.567153    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:24.875531 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:24.875426    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:24.939780 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:24.939742    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:25.191758 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:25.191666    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:25.233788 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:25.233758    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6st8d_d928b5bc-8033-4f03-be65-0989dabe8581/dns-node-resolver/0.log"
Jul 02 21:19:25.322698 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:25.322654    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:25.359398 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:25.359360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:25.359561 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:25.359524    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:25.359621 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:25.359609    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs podName:30cc5577-2894-4468-9c39-a462c823e44e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:57.359584324 +0000 UTC m=+66.093217617 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs") pod "network-metrics-daemon-fd5sb" (UID: "30cc5577-2894-4468-9c39-a462c823e44e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 21:19:25.460070 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:25.460042    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:25.460553 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:25.460147    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:25.460553 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:25.460199    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret podName:71429679-06c0-480b-8c79-08a2c5682ca1 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:57.460185715 +0000 UTC m=+66.193819002 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret") pod "global-pull-secret-syncer-mdqbk" (UID: "71429679-06c0-480b-8c79-08a2c5682ca1") : object "kube-system"/"original-pull-secret" not registered
Jul 02 21:19:25.560700 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:25.560665    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzt58\" (UniqueName: \"kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58\") pod \"network-check-target-pjcpf\" (UID: \"a723a652-5f75-4d27-acc7-5489c2aba607\") " pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:25.560865 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:25.560808    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 21:19:25.560865 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:25.560825    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 21:19:25.560865 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:25.560837    2564 projected.go:196] Error preparing data for projected volume kube-api-access-lzt58 for pod openshift-network-diagnostics/network-check-target-pjcpf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:25.561049 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:25.560899    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58 podName:a723a652-5f75-4d27-acc7-5489c2aba607 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:57.560884798 +0000 UTC m=+66.294518086 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lzt58" (UniqueName: "kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58") pod "network-check-target-pjcpf" (UID: "a723a652-5f75-4d27-acc7-5489c2aba607") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 21:19:25.832780 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:25.832750    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:25.832899 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:25.832867    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:25.832899 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:25.832877    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mdqbk" podUID="71429679-06c0-480b-8c79-08a2c5682ca1"
Jul 02 21:19:25.832899 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:25.832897    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:25.833094 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:25.832981    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd5sb" podUID="30cc5577-2894-4468-9c39-a462c823e44e"
Jul 02 21:19:25.833157 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:25.833093    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjcpf" podUID="a723a652-5f75-4d27-acc7-5489c2aba607"
Jul 02 21:19:26.109841 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.109754    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-121.ec2.internal" event="NodeReady"
Jul 02 21:19:26.110016 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.109919    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 21:19:26.158078 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.158031    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8f494dc48-hzdrn"]
Jul 02 21:19:26.188441 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.188411    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6xfl8"]
Jul 02 21:19:26.188609 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.188562    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.191493 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.191460    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 21:19:26.194924 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.194848    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 21:19:26.194924 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.194863    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 21:19:26.195985 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.195730    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p4wrm\""
Jul 02 21:19:26.200748 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.200728    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 21:19:26.204724 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.204682    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8f494dc48-hzdrn"]
Jul 02 21:19:26.204724 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.204722    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-948td"]
Jul 02 21:19:26.204885 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.204806    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6xfl8"
Jul 02 21:19:26.208940 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.208916    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bftgm\""
Jul 02 21:19:26.208940 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.208932    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 21:19:26.209337 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.209319    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 21:19:26.209437 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.209359    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 21:19:26.224650 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.224626    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6xfl8"]
Jul 02 21:19:26.224822 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.224665    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-948td"]
Jul 02 21:19:26.224822 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.224694    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f7sz7"]
Jul 02 21:19:26.224822 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.224704    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.227590 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.227567    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 21:19:26.227826 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.227807    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 21:19:26.227931 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.227897    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-69xqz\""
Jul 02 21:19:26.237562 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.237544    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g55b4_1c5ca752-f1cb-483c-9f16-7b425d081d7b/node-ca/0.log"
Jul 02 21:19:26.244740 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.244712    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f7sz7"]
Jul 02 21:19:26.244854 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.244837    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.249774 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.249750    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 21:19:26.250097 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.250077    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 21:19:26.250369 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.250353    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t9dsl\""
Jul 02 21:19:26.250700 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.250621    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 21:19:26.251240 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.251223    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 21:19:26.366936 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.366846    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.366936 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.366892    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-installation-pull-secrets\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.366936 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.366923    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65jw\" (UniqueName: \"kubernetes.io/projected/33841b4b-1329-4435-801f-3e2f803b298e-kube-api-access-w65jw\") pod \"ingress-canary-6xfl8\" (UID: \"33841b4b-1329-4435-801f-3e2f803b298e\") " pod="openshift-ingress-canary/ingress-canary-6xfl8"
Jul 02 21:19:26.367183 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.366973    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgjps\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-kube-api-access-xgjps\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.367183 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367059    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33841b4b-1329-4435-801f-3e2f803b298e-cert\") pod \"ingress-canary-6xfl8\" (UID: \"33841b4b-1329-4435-801f-3e2f803b298e\") " pod="openshift-ingress-canary/ingress-canary-6xfl8"
Jul 02 21:19:26.367183 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-crio-socket\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.367183 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367112    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.367183 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367145    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-config-volume\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.367183 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367162    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.367183 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367179    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-tmp-dir\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.367463 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367198    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-image-registry-private-configuration\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.367463 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367266    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-trusted-ca\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.367463 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-bound-sa-token\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.367463 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367309    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-data-volume\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.367463 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp92c\" (UniqueName: \"kubernetes.io/projected/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-kube-api-access-bp92c\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.367463 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367390    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529pm\" (UniqueName: \"kubernetes.io/projected/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-api-access-529pm\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.367463 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-ca-trust-extracted\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.367463 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367458    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.367785 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.367491    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-certificates\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.425052 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.425013    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:26.468200 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468171    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33841b4b-1329-4435-801f-3e2f803b298e-cert\") pod \"ingress-canary-6xfl8\" (UID: \"33841b4b-1329-4435-801f-3e2f803b298e\") " pod="openshift-ingress-canary/ingress-canary-6xfl8"
Jul 02 21:19:26.468200 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-crio-socket\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-config-volume\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468433    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-crio-socket\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468445    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-tmp-dir\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468473    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-image-registry-private-configuration\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.468445    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468527    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-trusted-ca\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.468562    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls podName:a6e7e14b-5fe7-4b9d-9edc-496ceb857389 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:26.968542033 +0000 UTC m=+35.702175319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f7sz7" (UID: "a6e7e14b-5fe7-4b9d-9edc-496ceb857389") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468584    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-bound-sa-token\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468612    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-data-volume\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468649    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp92c\" (UniqueName: \"kubernetes.io/projected/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-kube-api-access-bp92c\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468676    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529pm\" (UniqueName: \"kubernetes.io/projected/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-api-access-529pm\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.468725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468682    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-tmp-dir\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468705    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-ca-trust-extracted\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-certificates\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468808    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-installation-pull-secrets\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65jw\" (UniqueName: \"kubernetes.io/projected/33841b4b-1329-4435-801f-3e2f803b298e-kube-api-access-w65jw\") pod \"ingress-canary-6xfl8\" (UID: \"33841b4b-1329-4435-801f-3e2f803b298e\") " pod="openshift-ingress-canary/ingress-canary-6xfl8"
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.468499    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.469140    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm podName:a6e7e14b-5fe7-4b9d-9edc-496ceb857389 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:26.969124603 +0000 UTC m=+35.702757927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f7sz7" (UID: "a6e7e14b-5fe7-4b9d-9edc-496ceb857389") : configmap "kube-rbac-proxy" not found
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.469071    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-config-volume\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.468937    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjps\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-kube-api-access-xgjps\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.469244    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:26.469260 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.469263    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8f494dc48-hzdrn: secret "image-registry-tls" not found
Jul 02 21:19:26.469696 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.469309    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:26.469696 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.469450    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-trusted-ca\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.469696 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.469466    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-data-volume\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.469696 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.469314    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls podName:7314ec50-bab2-4b39-b051-cdd84c7b9d8e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:26.969297358 +0000 UTC m=+35.702930650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls") pod "image-registry-8f494dc48-hzdrn" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e") : secret "image-registry-tls" not found
Jul 02 21:19:26.469696 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.469504    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls podName:f32deb0f-16bc-480c-90c8-70fb7d8e81b6 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:26.969490931 +0000 UTC m=+35.703124218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls") pod "dns-default-948td" (UID: "f32deb0f-16bc-480c-90c8-70fb7d8e81b6") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:26.469892 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.469765    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-ca-trust-extracted\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.469970 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.469949    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-certificates\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.473270 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.473250    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-installation-pull-secrets\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.473270 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.473260    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-image-registry-private-configuration\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.473463 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.473405    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33841b4b-1329-4435-801f-3e2f803b298e-cert\") pod \"ingress-canary-6xfl8\" (UID: \"33841b4b-1329-4435-801f-3e2f803b298e\") " pod="openshift-ingress-canary/ingress-canary-6xfl8"
Jul 02 21:19:26.477924 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.477868    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-529pm\" (UniqueName: \"kubernetes.io/projected/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-api-access-529pm\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.479489 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.479464    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-bound-sa-token\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.479722 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.479697    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65jw\" (UniqueName: \"kubernetes.io/projected/33841b4b-1329-4435-801f-3e2f803b298e-kube-api-access-w65jw\") pod \"ingress-canary-6xfl8\" (UID: \"33841b4b-1329-4435-801f-3e2f803b298e\") " pod="openshift-ingress-canary/ingress-canary-6xfl8"
Jul 02 21:19:26.480348 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.480298    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjps\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-kube-api-access-xgjps\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.480981 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.480953    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp92c\" (UniqueName: \"kubernetes.io/projected/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-kube-api-access-bp92c\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.516869 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.516840    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6xfl8"
Jul 02 21:19:26.525443 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.525409    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:26.744951 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.744914    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-76tk4"]
Jul 02 21:19:26.766606 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.766580    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-76tk4"]
Jul 02 21:19:26.766754 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.766705    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-76tk4"
Jul 02 21:19:26.769458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.769431    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 02 21:19:26.769581 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.769431    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 02 21:19:26.769581 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.769522    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 02 21:19:26.769581 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.769531    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 02 21:19:26.769799 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.769780    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-png9s\""
Jul 02 21:19:26.788799 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.788770    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:26.841123 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.841086    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:26.871947 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.871912    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6094ed2e-ff5a-49db-a547-d69b4d6d60a6-signing-key\") pod \"service-ca-758f599c5-76tk4\" (UID: \"6094ed2e-ff5a-49db-a547-d69b4d6d60a6\") " pod="openshift-service-ca/service-ca-758f599c5-76tk4"
Jul 02 21:19:26.872141 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.872023    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6094ed2e-ff5a-49db-a547-d69b4d6d60a6-signing-cabundle\") pod \"service-ca-758f599c5-76tk4\" (UID: \"6094ed2e-ff5a-49db-a547-d69b4d6d60a6\") " pod="openshift-service-ca/service-ca-758f599c5-76tk4"
Jul 02 21:19:26.872141 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.872050    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d92z\" (UniqueName: \"kubernetes.io/projected/6094ed2e-ff5a-49db-a547-d69b4d6d60a6-kube-api-access-5d92z\") pod \"service-ca-758f599c5-76tk4\" (UID: \"6094ed2e-ff5a-49db-a547-d69b4d6d60a6\") " pod="openshift-service-ca/service-ca-758f599c5-76tk4"
Jul 02 21:19:26.972792 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.972758    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6094ed2e-ff5a-49db-a547-d69b4d6d60a6-signing-key\") pod \"service-ca-758f599c5-76tk4\" (UID: \"6094ed2e-ff5a-49db-a547-d69b4d6d60a6\") " pod="openshift-service-ca/service-ca-758f599c5-76tk4"
Jul 02 21:19:26.972971 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.972822    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:26.972971 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.972862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:26.972971 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.972890    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6094ed2e-ff5a-49db-a547-d69b4d6d60a6-signing-cabundle\") pod \"service-ca-758f599c5-76tk4\" (UID: \"6094ed2e-ff5a-49db-a547-d69b4d6d60a6\") " pod="openshift-service-ca/service-ca-758f599c5-76tk4"
Jul 02 21:19:26.972971 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.972934    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:26.972971 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.972954    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8f494dc48-hzdrn: secret "image-registry-tls" not found
Jul 02 21:19:26.973236 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.973028    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:26.973236 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.973039    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls podName:7314ec50-bab2-4b39-b051-cdd84c7b9d8e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:27.973018094 +0000 UTC m=+36.706651403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls") pod "image-registry-8f494dc48-hzdrn" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e") : secret "image-registry-tls" not found
Jul 02 21:19:26.973236 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.973077    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls podName:f32deb0f-16bc-480c-90c8-70fb7d8e81b6 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:27.973062699 +0000 UTC m=+36.706696002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls") pod "dns-default-948td" (UID: "f32deb0f-16bc-480c-90c8-70fb7d8e81b6") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:26.973236 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.973106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d92z\" (UniqueName: \"kubernetes.io/projected/6094ed2e-ff5a-49db-a547-d69b4d6d60a6-kube-api-access-5d92z\") pod \"service-ca-758f599c5-76tk4\" (UID: \"6094ed2e-ff5a-49db-a547-d69b4d6d60a6\") " pod="openshift-service-ca/service-ca-758f599c5-76tk4"
Jul 02 21:19:26.973236 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.973154    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.973236 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.973183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:26.973491 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.973328    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:26.973491 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.973359    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls podName:a6e7e14b-5fe7-4b9d-9edc-496ceb857389 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:27.973350464 +0000 UTC m=+36.706983751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f7sz7" (UID: "a6e7e14b-5fe7-4b9d-9edc-496ceb857389") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:26.973491 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.973449    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:19:26.973491 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:26.973478    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm podName:a6e7e14b-5fe7-4b9d-9edc-496ceb857389 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:27.973470998 +0000 UTC m=+36.707104283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f7sz7" (UID: "a6e7e14b-5fe7-4b9d-9edc-496ceb857389") : configmap "kube-rbac-proxy" not found
Jul 02 21:19:26.973681 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.973657    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6094ed2e-ff5a-49db-a547-d69b4d6d60a6-signing-cabundle\") pod \"service-ca-758f599c5-76tk4\" (UID: \"6094ed2e-ff5a-49db-a547-d69b4d6d60a6\") " pod="openshift-service-ca/service-ca-758f599c5-76tk4"
Jul 02 21:19:26.976036 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.976014    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6094ed2e-ff5a-49db-a547-d69b4d6d60a6-signing-key\") pod \"service-ca-758f599c5-76tk4\" (UID: \"6094ed2e-ff5a-49db-a547-d69b4d6d60a6\") " pod="openshift-service-ca/service-ca-758f599c5-76tk4"
Jul 02 21:19:26.981200 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.981181    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d92z\" (UniqueName: \"kubernetes.io/projected/6094ed2e-ff5a-49db-a547-d69b4d6d60a6-kube-api-access-5d92z\") pod \"service-ca-758f599c5-76tk4\" (UID: \"6094ed2e-ff5a-49db-a547-d69b4d6d60a6\") " pod="openshift-service-ca/service-ca-758f599c5-76tk4"
Jul 02 21:19:26.999626 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:26.999568    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:27.077529 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.077500    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-76tk4"
Jul 02 21:19:27.100494 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.100466    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:27.213890 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.213861    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6xfl8"]
Jul 02 21:19:27.216788 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.216764    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-76tk4"]
Jul 02 21:19:27.278589 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:19:27.278515    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33841b4b_1329_4435_801f_3e2f803b298e.slice/crio-69087b0cc8456ec65a581ee818e9b467db08a9d980681a5c6f191f4adaf7699d WatchSource:0}: Error finding container 69087b0cc8456ec65a581ee818e9b467db08a9d980681a5c6f191f4adaf7699d: Status 404 returned error can't find the container with id 69087b0cc8456ec65a581ee818e9b467db08a9d980681a5c6f191f4adaf7699d
Jul 02 21:19:27.279209 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:19:27.279187    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6094ed2e_ff5a_49db_a547_d69b4d6d60a6.slice/crio-4b14e6840c9df2acf18318a5613014e7f0b1cc573a8fc07b4c8946559b590d12 WatchSource:0}: Error finding container 4b14e6840c9df2acf18318a5613014e7f0b1cc573a8fc07b4c8946559b590d12: Status 404 returned error can't find the container with id 4b14e6840c9df2acf18318a5613014e7f0b1cc573a8fc07b4c8946559b590d12
Jul 02 21:19:27.319208 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.319173    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:27.794118 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.794057    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:27.829074 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.829037    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:27.829435 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.829396    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:27.829850 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.829608    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:27.832801 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.832775    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 21:19:27.834453 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.834071    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lxff2\""
Jul 02 21:19:27.834453 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.834180    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r4vsj\""
Jul 02 21:19:27.834453 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.834218    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 21:19:27.834453 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.834290    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 21:19:27.835136 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.835116    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 21:19:27.981814 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.981777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:27.982020 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.981894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:27.982020 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.981930    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:27.982020 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:27.981949    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:27.982020 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:27.982008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:27.982210 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:27.982039    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls podName:f32deb0f-16bc-480c-90c8-70fb7d8e81b6 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:29.98201888 +0000 UTC m=+38.715652165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls") pod "dns-default-948td" (UID: "f32deb0f-16bc-480c-90c8-70fb7d8e81b6") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:27.982210 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:27.982097    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:27.982210 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:27.982108    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8f494dc48-hzdrn: secret "image-registry-tls" not found
Jul 02 21:19:27.982365 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:27.982144    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls podName:7314ec50-bab2-4b39-b051-cdd84c7b9d8e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:29.98213294 +0000 UTC m=+38.715766230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls") pod "image-registry-8f494dc48-hzdrn" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e") : secret "image-registry-tls" not found
Jul 02 21:19:27.982365 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:27.982283    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:19:27.982365 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:27.982356    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm podName:a6e7e14b-5fe7-4b9d-9edc-496ceb857389 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:29.982347995 +0000 UTC m=+38.715981285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f7sz7" (UID: "a6e7e14b-5fe7-4b9d-9edc-496ceb857389") : configmap "kube-rbac-proxy" not found
Jul 02 21:19:27.982365 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:27.982290    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:27.982644 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:27.982392    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls podName:a6e7e14b-5fe7-4b9d-9edc-496ceb857389 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:29.982385842 +0000 UTC m=+38.716019131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f7sz7" (UID: "a6e7e14b-5fe7-4b9d-9edc-496ceb857389") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:28.021656 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:28.021625    2564 generic.go:356] "Generic (PLEG): container finished" podID="247b8f4d-bbc5-44b7-981b-aa42c30c031f" containerID="5890ebe006b79069d7715845ed5df61eda0f1aa68e76a0ee313d53bd48bcfe52" exitCode=0
Jul 02 21:19:28.021807 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:28.021721    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdmdj" event={"ID":"247b8f4d-bbc5-44b7-981b-aa42c30c031f","Type":"ContainerDied","Data":"5890ebe006b79069d7715845ed5df61eda0f1aa68e76a0ee313d53bd48bcfe52"}
Jul 02 21:19:28.023413 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:28.023390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6xfl8" event={"ID":"33841b4b-1329-4435-801f-3e2f803b298e","Type":"ContainerStarted","Data":"69087b0cc8456ec65a581ee818e9b467db08a9d980681a5c6f191f4adaf7699d"}
Jul 02 21:19:28.025017 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:28.024977    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-76tk4" event={"ID":"6094ed2e-ff5a-49db-a547-d69b4d6d60a6","Type":"ContainerStarted","Data":"4b14e6840c9df2acf18318a5613014e7f0b1cc573a8fc07b4c8946559b590d12"}
Jul 02 21:19:29.031493 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:29.031458    2564 generic.go:356] "Generic (PLEG): container finished" podID="247b8f4d-bbc5-44b7-981b-aa42c30c031f" containerID="5c3140acd0768f4d5e790d476059256022f197edf2ad7ed20774668457607be1" exitCode=0
Jul 02 21:19:29.031894 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:29.031510    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdmdj" event={"ID":"247b8f4d-bbc5-44b7-981b-aa42c30c031f","Type":"ContainerDied","Data":"5c3140acd0768f4d5e790d476059256022f197edf2ad7ed20774668457607be1"}
Jul 02 21:19:30.002232 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:30.002199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:30.002314 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:30.002251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:30.002352 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:30.002326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:30.002388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:30.002359    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:30.002531 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:30.002513    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:30.002615 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:30.002530    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:19:30.002615 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:30.002567    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls podName:f32deb0f-16bc-480c-90c8-70fb7d8e81b6 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:34.002549727 +0000 UTC m=+42.736183017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls") pod "dns-default-948td" (UID: "f32deb0f-16bc-480c-90c8-70fb7d8e81b6") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:30.002615 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:30.002592    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm podName:a6e7e14b-5fe7-4b9d-9edc-496ceb857389 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:34.002574365 +0000 UTC m=+42.736207665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f7sz7" (UID: "a6e7e14b-5fe7-4b9d-9edc-496ceb857389") : configmap "kube-rbac-proxy" not found
Jul 02 21:19:30.002767 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:30.002637    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:30.002767 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:30.002649    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8f494dc48-hzdrn: secret "image-registry-tls" not found
Jul 02 21:19:30.002767 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:30.002675    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:30.002767 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:30.002682    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls podName:7314ec50-bab2-4b39-b051-cdd84c7b9d8e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:34.002671066 +0000 UTC m=+42.736304357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls") pod "image-registry-8f494dc48-hzdrn" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e") : secret "image-registry-tls" not found
Jul 02 21:19:30.002767 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:30.002714    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls podName:a6e7e14b-5fe7-4b9d-9edc-496ceb857389 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:34.002701874 +0000 UTC m=+42.736335166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f7sz7" (UID: "a6e7e14b-5fe7-4b9d-9edc-496ceb857389") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:30.038461 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:30.038407    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdmdj" event={"ID":"247b8f4d-bbc5-44b7-981b-aa42c30c031f","Type":"ContainerStarted","Data":"c047dabd13441333024dfcf4b02e12d46a946a861743fc47260533a2cfee9eb6"}
Jul 02 21:19:30.041218 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:30.040161    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6xfl8" event={"ID":"33841b4b-1329-4435-801f-3e2f803b298e","Type":"ContainerStarted","Data":"67e09e14f4467e10297fc540f3bf7d6d059075b76af2a16d6aedfa39dafd8c4a"}
Jul 02 21:19:30.042658 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:30.042379    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-76tk4" event={"ID":"6094ed2e-ff5a-49db-a547-d69b4d6d60a6","Type":"ContainerStarted","Data":"6406841a0c84082f93c8d921de6a1d67f602c1292da63d8b8935f015a3424624"}
Jul 02 21:19:30.080354 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:30.080296    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-76tk4" podStartSLOduration=1.456909094 podStartE2EDuration="4.080281492s" totalImagesPullingTime="2.623372398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:30.079635566 +0000 UTC m=+38.813268875" watchObservedRunningTime="2026-07-02 21:19:30.080281492 +0000 UTC m=+38.813914799"
Jul 02 21:19:30.080533 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:30.080514    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wdmdj" podStartSLOduration=9.663351605999999 podStartE2EDuration="39.080509106s" totalImagesPullingTime="29.4171575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:30.06406566 +0000 UTC m=+38.797698993" watchObservedRunningTime="2026-07-02 21:19:30.080509106 +0000 UTC m=+38.814142392"
Jul 02 21:19:30.095644 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:30.095574    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6xfl8" podStartSLOduration=1.473103027 podStartE2EDuration="4.095552528s" totalImagesPullingTime="2.622449501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:30.094451147 +0000 UTC m=+38.828084453" watchObservedRunningTime="2026-07-02 21:19:30.095552528 +0000 UTC m=+38.829185839"
Jul 02 21:19:30.883427 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:30.883392    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:31.001706 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:31.001675    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:31.094213 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:31.094180    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:31.242985 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:31.242939    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:31.397745 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:31.397700    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:31.602494 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:31.602406    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:31.704419 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:31.704379    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:31.804505 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:31.804462    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:31.910908 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:31.910809    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:32.595663 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:32.595628    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:33.008015 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:33.007964    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:33.456982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:33.456929    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:33.618399 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:33.618365    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:34.031771 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:34.031730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:34.031965 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:34.031781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:34.031965 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:34.031848    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 21:19:34.031965 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:34.031877    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:34.031965 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:34.031910    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm podName:a6e7e14b-5fe7-4b9d-9edc-496ceb857389 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:42.031895817 +0000 UTC m=+50.765529108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f7sz7" (UID: "a6e7e14b-5fe7-4b9d-9edc-496ceb857389") : configmap "kube-rbac-proxy" not found
Jul 02 21:19:34.031965 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:34.031855    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:34.031965 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:34.031932    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 21:19:34.031965 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:34.031947    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8f494dc48-hzdrn: secret "image-registry-tls" not found
Jul 02 21:19:34.032256 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:34.031934    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls podName:a6e7e14b-5fe7-4b9d-9edc-496ceb857389 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:42.03191779 +0000 UTC m=+50.765551089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f7sz7" (UID: "a6e7e14b-5fe7-4b9d-9edc-496ceb857389") : secret "insights-runtime-extractor-tls" not found
Jul 02 21:19:34.032256 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:34.032064    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls podName:7314ec50-bab2-4b39-b051-cdd84c7b9d8e nodeName:}" failed. No retries permitted until 2026-07-02 21:19:42.031987356 +0000 UTC m=+50.765620646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls") pod "image-registry-8f494dc48-hzdrn" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e") : secret "image-registry-tls" not found
Jul 02 21:19:34.032256 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:34.032106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:34.032256 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:34.032199    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 21:19:34.032256 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:19:34.032235    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls podName:f32deb0f-16bc-480c-90c8-70fb7d8e81b6 nodeName:}" failed. No retries permitted until 2026-07-02 21:19:42.032225397 +0000 UTC m=+50.765858686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls") pod "dns-default-948td" (UID: "f32deb0f-16bc-480c-90c8-70fb7d8e81b6") : secret "dns-default-metrics-tls" not found
Jul 02 21:19:34.208694 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:34.208662    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:34.383085 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:34.382980    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:34.627285 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:34.627252    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:34.800027 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:34.799963    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:35.104199 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:35.104110    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:35.386868 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:35.386777    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:35.779735 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:35.779694    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:35.836065 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:35.836034    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:35.981872 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:35.981837    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:36.583803 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:36.583766    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:36.985580 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:36.985549    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:37.231878 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:37.231846    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:37.304315 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:37.304226    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:37.387008 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:37.386970    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:37.586410 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:37.586314    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:37.995224 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:37.995192    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:38.585824 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:38.585791    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:39.186198 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:39.186167    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:39.380098 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:39.380067    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:39.446945 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:39.446911    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:39.790011 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:39.789909    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:39.986260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:39.986225    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:40.398185 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:40.398147    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:41.186546 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:41.186516    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:41.408849 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:41.408818    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:41.594885 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:41.594849    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:41.805573 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:41.805535    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:42.097861 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.097831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:42.098069 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.097899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:42.098069 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.097931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:42.098069 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.097979    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:42.098523 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.098498    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:42.101457 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.101426    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32deb0f-16bc-480c-90c8-70fb7d8e81b6-metrics-tls\") pod \"dns-default-948td\" (UID: \"f32deb0f-16bc-480c-90c8-70fb7d8e81b6\") " pod="openshift-dns/dns-default-948td"
Jul 02 21:19:42.101579 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.101481    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls\") pod \"image-registry-8f494dc48-hzdrn\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") " pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:42.101579 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.101494    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6e7e14b-5fe7-4b9d-9edc-496ceb857389-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7sz7\" (UID: \"a6e7e14b-5fe7-4b9d-9edc-496ceb857389\") " pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:42.134497 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.134469    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-948td"
Jul 02 21:19:42.155666 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.155646    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f7sz7"
Jul 02 21:19:42.281932 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.281906    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-948td"]
Jul 02 21:19:42.283931 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:19:42.283902    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32deb0f_16bc_480c_90c8_70fb7d8e81b6.slice/crio-7bc8519e9742c2e26f65787540e2d0f02144db0929f4bf270e70ab9877ca220e WatchSource:0}: Error finding container 7bc8519e9742c2e26f65787540e2d0f02144db0929f4bf270e70ab9877ca220e: Status 404 returned error can't find the container with id 7bc8519e9742c2e26f65787540e2d0f02144db0929f4bf270e70ab9877ca220e
Jul 02 21:19:42.305368 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.305343    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f7sz7"]
Jul 02 21:19:42.307119 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:19:42.307094    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e7e14b_5fe7_4b9d_9edc_496ceb857389.slice/crio-e54612a1d05dfa85b62c54eebd4c01caca6e983dccdec0d5cd53e22f942f8e7d WatchSource:0}: Error finding container e54612a1d05dfa85b62c54eebd4c01caca6e983dccdec0d5cd53e22f942f8e7d: Status 404 returned error can't find the container with id e54612a1d05dfa85b62c54eebd4c01caca6e983dccdec0d5cd53e22f942f8e7d
Jul 02 21:19:42.399963 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.399870    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:42.570339 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:42.570303    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8f494dc48-hzdrn"]
Jul 02 21:19:42.573700 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:19:42.573673    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7314ec50_bab2_4b39_b051_cdd84c7b9d8e.slice/crio-7cb307b5f61ca9a9bf3651cdfa866fa4570d907b574040aa4309f98580735b4b WatchSource:0}: Error finding container 7cb307b5f61ca9a9bf3651cdfa866fa4570d907b574040aa4309f98580735b4b: Status 404 returned error can't find the container with id 7cb307b5f61ca9a9bf3651cdfa866fa4570d907b574040aa4309f98580735b4b
Jul 02 21:19:43.080577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:43.079740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn" event={"ID":"7314ec50-bab2-4b39-b051-cdd84c7b9d8e","Type":"ContainerStarted","Data":"d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85"}
Jul 02 21:19:43.080577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:43.079803    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn" event={"ID":"7314ec50-bab2-4b39-b051-cdd84c7b9d8e","Type":"ContainerStarted","Data":"7cb307b5f61ca9a9bf3651cdfa866fa4570d907b574040aa4309f98580735b4b"}
Jul 02 21:19:43.080577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:43.080531    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:19:43.083050 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:43.082971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f7sz7" event={"ID":"a6e7e14b-5fe7-4b9d-9edc-496ceb857389","Type":"ContainerStarted","Data":"e54612a1d05dfa85b62c54eebd4c01caca6e983dccdec0d5cd53e22f942f8e7d"}
Jul 02 21:19:43.086164 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:43.086096    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-948td" event={"ID":"f32deb0f-16bc-480c-90c8-70fb7d8e81b6","Type":"ContainerStarted","Data":"7bc8519e9742c2e26f65787540e2d0f02144db0929f4bf270e70ab9877ca220e"}
Jul 02 21:19:43.100328 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:43.100273    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn" podStartSLOduration=25.100254818 podStartE2EDuration="25.100254818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:43.099757295 +0000 UTC m=+51.833390603" watchObservedRunningTime="2026-07-02 21:19:43.100254818 +0000 UTC m=+51.833888130"
Jul 02 21:19:46.854846 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:46.854768    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-shwvz"]
Jul 02 21:19:46.857753 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:46.857738    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-shwvz"
Jul 02 21:19:46.860393 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:46.860369    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 02 21:19:46.861440 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:46.861418    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hwdbm\""
Jul 02 21:19:46.861539 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:46.861418    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 02 21:19:46.867324 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:46.867302    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-shwvz"]
Jul 02 21:19:46.935101 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:46.935068    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/287c4ada-4462-484f-9832-b617173a665f-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-shwvz\" (UID: \"287c4ada-4462-484f-9832-b617173a665f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-shwvz"
Jul 02 21:19:46.935260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:46.935155    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/287c4ada-4462-484f-9832-b617173a665f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-shwvz\" (UID: \"287c4ada-4462-484f-9832-b617173a665f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-shwvz"
Jul 02 21:19:46.939618 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:46.939596    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8f494dc48-hzdrn"]
Jul 02 21:19:46.982361 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:46.982317    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:47.007631 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.007601    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f445984c5-q9n6l"]
Jul 02 21:19:47.010423 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.010407    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.027268 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.027241    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f445984c5-q9n6l"]
Jul 02 21:19:47.035675 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.035651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-registry-certificates\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.035776 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.035682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-trusted-ca\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.035776 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.035723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-installation-pull-secrets\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.035848 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.035797    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-image-registry-private-configuration\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.035848 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.035820    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-bound-sa-token\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.035906 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.035854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/287c4ada-4462-484f-9832-b617173a665f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-shwvz\" (UID: \"287c4ada-4462-484f-9832-b617173a665f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-shwvz"
Jul 02 21:19:47.035906 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.035871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/287c4ada-4462-484f-9832-b617173a665f-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-shwvz\" (UID: \"287c4ada-4462-484f-9832-b617173a665f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-shwvz"
Jul 02 21:19:47.035906 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.035895    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-ca-trust-extracted\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.036055 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.036037    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68h7m\" (UniqueName: \"kubernetes.io/projected/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-kube-api-access-68h7m\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.036113 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.036080    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-registry-tls\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.036553 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.036537    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/287c4ada-4462-484f-9832-b617173a665f-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-shwvz\" (UID: \"287c4ada-4462-484f-9832-b617173a665f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-shwvz"
Jul 02 21:19:47.038255 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.038236    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/287c4ada-4462-484f-9832-b617173a665f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-shwvz\" (UID: \"287c4ada-4462-484f-9832-b617173a665f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-shwvz"
Jul 02 21:19:47.099868 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.099831    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f7sz7" event={"ID":"a6e7e14b-5fe7-4b9d-9edc-496ceb857389","Type":"ContainerStarted","Data":"8e77ada8c72bf56bf8c32262d01374edc32e9507b555075478824e080b459288"}
Jul 02 21:19:47.101891 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.101862    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-948td" event={"ID":"f32deb0f-16bc-480c-90c8-70fb7d8e81b6","Type":"ContainerStarted","Data":"ac503c9831fb490c709997c6fc32e72343c496bd228ccb556300946adfe9db28"}
Jul 02 21:19:47.102037 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.101909    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-948td" event={"ID":"f32deb0f-16bc-480c-90c8-70fb7d8e81b6","Type":"ContainerStarted","Data":"791a19831cee304861b7847f853cf18bd60d7df48814ddeadb8f051f56b0b975"}
Jul 02 21:19:47.102037 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.102026    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-948td"
Jul 02 21:19:47.125886 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.125794    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-948td" podStartSLOduration=16.882406247 podStartE2EDuration="21.125774934s" totalImagesPullingTime="4.243368687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:47.124820888 +0000 UTC m=+55.858454223" watchObservedRunningTime="2026-07-02 21:19:47.125774934 +0000 UTC m=+55.859408262"
Jul 02 21:19:47.130567 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.130546    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:47.137040 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.137021    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-registry-certificates\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.137148 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.137059    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-trusted-ca\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.137148 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.137119    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-installation-pull-secrets\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.137260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.137173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-image-registry-private-configuration\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.137260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.137197    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-bound-sa-token\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.137260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.137242    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-ca-trust-extracted\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.137388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.137272    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68h7m\" (UniqueName: \"kubernetes.io/projected/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-kube-api-access-68h7m\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.137388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.137297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-registry-tls\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.137964 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.137887    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-registry-certificates\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.137964 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.137891    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-ca-trust-extracted\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.138335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.138312    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-trusted-ca\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.139823 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.139795    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-registry-tls\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.139917 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.139806    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-installation-pull-secrets\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.139917 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.139875    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-image-registry-private-configuration\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.152275 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.152246    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-bound-sa-token\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.153311 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.153296    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68h7m\" (UniqueName: \"kubernetes.io/projected/0bebaf20-3e91-4341-b6c2-498ab54f5fdd-kube-api-access-68h7m\") pod \"image-registry-6f445984c5-q9n6l\" (UID: \"0bebaf20-3e91-4341-b6c2-498ab54f5fdd\") " pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.167090 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.167070    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-shwvz"
Jul 02 21:19:47.217492 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.217451    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:47.318799 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.318766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:47.346421 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.346392    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-shwvz"]
Jul 02 21:19:47.349013 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:19:47.348959    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287c4ada_4462_484f_9832_b617173a665f.slice/crio-43a24f54a321b6a77b142b85df10eb8c1faacdec3a3d8d0247790f5aacb668cb WatchSource:0}: Error finding container 43a24f54a321b6a77b142b85df10eb8c1faacdec3a3d8d0247790f5aacb668cb: Status 404 returned error can't find the container with id 43a24f54a321b6a77b142b85df10eb8c1faacdec3a3d8d0247790f5aacb668cb
Jul 02 21:19:47.444400 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:47.444378    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f445984c5-q9n6l"]
Jul 02 21:19:47.446510 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:19:47.446487    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bebaf20_3e91_4341_b6c2_498ab54f5fdd.slice/crio-c10cbdf050ae5c0fa7397d66e6aa99dc26d2b3619ba40d5c95e2bbd2ee81214e WatchSource:0}: Error finding container c10cbdf050ae5c0fa7397d66e6aa99dc26d2b3619ba40d5c95e2bbd2ee81214e: Status 404 returned error can't find the container with id c10cbdf050ae5c0fa7397d66e6aa99dc26d2b3619ba40d5c95e2bbd2ee81214e
Jul 02 21:19:48.107821 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:48.107774    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f445984c5-q9n6l" event={"ID":"0bebaf20-3e91-4341-b6c2-498ab54f5fdd","Type":"ContainerStarted","Data":"86741745ba3c77da90cac23e2b24986ccc530c86f49aabec8d994bd69c6f4d25"}
Jul 02 21:19:48.108287 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:48.107830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f445984c5-q9n6l" event={"ID":"0bebaf20-3e91-4341-b6c2-498ab54f5fdd","Type":"ContainerStarted","Data":"c10cbdf050ae5c0fa7397d66e6aa99dc26d2b3619ba40d5c95e2bbd2ee81214e"}
Jul 02 21:19:48.108287 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:48.107898    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:19:48.109318 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:48.109278    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-shwvz" event={"ID":"287c4ada-4462-484f-9832-b617173a665f","Type":"ContainerStarted","Data":"43a24f54a321b6a77b142b85df10eb8c1faacdec3a3d8d0247790f5aacb668cb"}
Jul 02 21:19:48.111352 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:48.111328    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f7sz7" event={"ID":"a6e7e14b-5fe7-4b9d-9edc-496ceb857389","Type":"ContainerStarted","Data":"1e962443c04ff8cc5cd1bcc4311ac12229dde1b120985fab065fae8203dc544f"}
Jul 02 21:19:48.131826 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:48.131752    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f445984c5-q9n6l" podStartSLOduration=2.131730686 podStartE2EDuration="2.131730686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:48.130668193 +0000 UTC m=+56.864301517" watchObservedRunningTime="2026-07-02 21:19:48.131730686 +0000 UTC m=+56.865363995"
Jul 02 21:19:48.602942 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:48.602905    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:48.661735 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:48.661646    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:49.116876 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:49.116831    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-shwvz" event={"ID":"287c4ada-4462-484f-9832-b617173a665f","Type":"ContainerStarted","Data":"c3993e4c1b1026d84e323b6d64c390eb2b074ca440e17161b7b749d7c9c08ac2"}
Jul 02 21:19:49.134150 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:49.134093    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-shwvz" podStartSLOduration=2.01395787 podStartE2EDuration="3.134070926s" totalImagesPullingTime="1.120113056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:49.132729741 +0000 UTC m=+57.866363083" watchObservedRunningTime="2026-07-02 21:19:49.134070926 +0000 UTC m=+57.867704234"
Jul 02 21:19:49.269056 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:49.269018    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:49.350235 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:49.350193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:51.124707 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:51.124666    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f7sz7" event={"ID":"a6e7e14b-5fe7-4b9d-9edc-496ceb857389","Type":"ContainerStarted","Data":"d700120b90f9da6d40a30c4917c4c2eaaec4a05ca70af781cded5c234c78ced9"}
Jul 02 21:19:51.145263 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:51.145210    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f7sz7" podStartSLOduration=17.289196148 podStartE2EDuration="25.145192822s" totalImagesPullingTime="7.855996674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:19:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:51.143563868 +0000 UTC m=+59.877197187" watchObservedRunningTime="2026-07-02 21:19:51.145192822 +0000 UTC m=+59.878826111"
Jul 02 21:19:52.011167 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.011138    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cd8lj"
Jul 02 21:19:52.188058 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.188021    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:52.243301 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.243270    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:52.674412 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.674376    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:52.777952 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.777919    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:52.853282 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.853251    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl"]
Jul 02 21:19:52.853745 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.853718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:52.857690 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.857671    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl"
Jul 02 21:19:52.860681 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.860658    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ktds6\""
Jul 02 21:19:52.860780 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.860681    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 02 21:19:52.866611 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.866589    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl"]
Jul 02 21:19:52.990085 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:52.990039    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a7dd804b-09c4-4d04-b3c4-ec3f1fdaef06-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-rtdsl\" (UID: \"a7dd804b-09c4-4d04-b3c4-ec3f1fdaef06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl"
Jul 02 21:19:53.091120 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:53.091082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a7dd804b-09c4-4d04-b3c4-ec3f1fdaef06-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-rtdsl\" (UID: \"a7dd804b-09c4-4d04-b3c4-ec3f1fdaef06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl"
Jul 02 21:19:53.093423 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:53.093392    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a7dd804b-09c4-4d04-b3c4-ec3f1fdaef06-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-rtdsl\" (UID: \"a7dd804b-09c4-4d04-b3c4-ec3f1fdaef06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl"
Jul 02 21:19:53.109218 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:53.109194    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:53.167442 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:53.167410    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl"
Jul 02 21:19:53.217663 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:53.217633    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:53.290407 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:53.290383    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl"]
Jul 02 21:19:53.292667 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:19:53.292639    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7dd804b_09c4_4d04_b3c4_ec3f1fdaef06.slice/crio-59122bffb0d85e5789d6d95a9c64f7ac5c8e015b4f621195d7fe136a041c8889 WatchSource:0}: Error finding container 59122bffb0d85e5789d6d95a9c64f7ac5c8e015b4f621195d7fe136a041c8889: Status 404 returned error can't find the container with id 59122bffb0d85e5789d6d95a9c64f7ac5c8e015b4f621195d7fe136a041c8889
Jul 02 21:19:53.298912 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:53.298887    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:54.134763 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:54.134719    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl" event={"ID":"a7dd804b-09c4-4d04-b3c4-ec3f1fdaef06","Type":"ContainerStarted","Data":"59122bffb0d85e5789d6d95a9c64f7ac5c8e015b4f621195d7fe136a041c8889"}
Jul 02 21:19:54.857173 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:54.857136    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:55.139182 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:55.139103    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl" event={"ID":"a7dd804b-09c4-4d04-b3c4-ec3f1fdaef06","Type":"ContainerStarted","Data":"79f463172c3a386c2e0d1a137d0cf75066cc5d6332181498f42a8f9c069a6c82"}
Jul 02 21:19:55.139316 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:55.139300    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl"
Jul 02 21:19:55.143781 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:55.143748    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl"
Jul 02 21:19:55.169805 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:55.169760    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rtdsl" podStartSLOduration=2.198294949 podStartE2EDuration="3.169746958s" totalImagesPullingTime="971.452009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:55.169728157 +0000 UTC m=+63.903361492" watchObservedRunningTime="2026-07-02 21:19:55.169746958 +0000 UTC m=+63.903380265"
Jul 02 21:19:55.887618 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:55.887586    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:55.991632 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:55.991602    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:56.124274 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:56.124244    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:57.114154 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.114126    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-948td"
Jul 02 21:19:57.427822 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.427732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:57.430853 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.430831    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 21:19:57.440222 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.440198    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cc5577-2894-4468-9c39-a462c823e44e-metrics-certs\") pod \"network-metrics-daemon-fd5sb\" (UID: \"30cc5577-2894-4468-9c39-a462c823e44e\") " pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:57.528244 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.528208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:57.531016 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.530982    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 21:19:57.541430 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.541410    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71429679-06c0-480b-8c79-08a2c5682ca1-original-pull-secret\") pod \"global-pull-secret-syncer-mdqbk\" (UID: \"71429679-06c0-480b-8c79-08a2c5682ca1\") " pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:57.551291 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.551272    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lxff2\""
Jul 02 21:19:57.558780 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.558760    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd5sb"
Jul 02 21:19:57.562987 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.562974    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mdqbk"
Jul 02 21:19:57.628952 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.628920    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzt58\" (UniqueName: \"kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58\") pod \"network-check-target-pjcpf\" (UID: \"a723a652-5f75-4d27-acc7-5489c2aba607\") " pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:57.632226 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.632201    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 21:19:57.642486 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.642468    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 21:19:57.653143 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.653117    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzt58\" (UniqueName: \"kubernetes.io/projected/a723a652-5f75-4d27-acc7-5489c2aba607-kube-api-access-lzt58\") pod \"network-check-target-pjcpf\" (UID: \"a723a652-5f75-4d27-acc7-5489c2aba607\") " pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:57.712900 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.712870    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fd5sb"]
Jul 02 21:19:57.716237 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:19:57.716209    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cc5577_2894_4468_9c39_a462c823e44e.slice/crio-d91f96991f8713c1267a002207e959978c8dbed1021c5b6f18c6c1b42b596b1b WatchSource:0}: Error finding container d91f96991f8713c1267a002207e959978c8dbed1021c5b6f18c6c1b42b596b1b: Status 404 returned error can't find the container with id d91f96991f8713c1267a002207e959978c8dbed1021c5b6f18c6c1b42b596b1b
Jul 02 21:19:57.729892 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.729859    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mdqbk"]
Jul 02 21:19:57.731783 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:19:57.731762    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71429679_06c0_480b_8c79_08a2c5682ca1.slice/crio-96b465859bc1597a9555ae9fa4013e5f029e1c651a5bc5f2801369d1b01ab563 WatchSource:0}: Error finding container 96b465859bc1597a9555ae9fa4013e5f029e1c651a5bc5f2801369d1b01ab563: Status 404 returned error can't find the container with id 96b465859bc1597a9555ae9fa4013e5f029e1c651a5bc5f2801369d1b01ab563
Jul 02 21:19:57.857934 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.857904    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r4vsj\""
Jul 02 21:19:57.865601 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:57.865578    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:19:58.017271 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:58.017237    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pjcpf"]
Jul 02 21:19:58.020187 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:19:58.020161    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda723a652_5f75_4d27_acc7_5489c2aba607.slice/crio-c808fd5df2b815950ba0504bcf121e3437c9ba65d442ed8ac5561e31429636d2 WatchSource:0}: Error finding container c808fd5df2b815950ba0504bcf121e3437c9ba65d442ed8ac5561e31429636d2: Status 404 returned error can't find the container with id c808fd5df2b815950ba0504bcf121e3437c9ba65d442ed8ac5561e31429636d2
Jul 02 21:19:58.149489 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:58.149450    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pjcpf" event={"ID":"a723a652-5f75-4d27-acc7-5489c2aba607","Type":"ContainerStarted","Data":"c808fd5df2b815950ba0504bcf121e3437c9ba65d442ed8ac5561e31429636d2"}
Jul 02 21:19:58.150545 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:58.150522    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fd5sb" event={"ID":"30cc5577-2894-4468-9c39-a462c823e44e","Type":"ContainerStarted","Data":"d91f96991f8713c1267a002207e959978c8dbed1021c5b6f18c6c1b42b596b1b"}
Jul 02 21:19:58.151461 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:58.151442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mdqbk" event={"ID":"71429679-06c0-480b-8c79-08a2c5682ca1","Type":"ContainerStarted","Data":"96b465859bc1597a9555ae9fa4013e5f029e1c651a5bc5f2801369d1b01ab563"}
Jul 02 21:19:59.159273 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:59.159186    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fd5sb" event={"ID":"30cc5577-2894-4468-9c39-a462c823e44e","Type":"ContainerStarted","Data":"c2179632cebe21407e6561e8d9e41d88d90f3fd442829a011961bbc6f1675227"}
Jul 02 21:19:59.159273 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:59.159247    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fd5sb" event={"ID":"30cc5577-2894-4468-9c39-a462c823e44e","Type":"ContainerStarted","Data":"a797d2930725daa897ceea1289d46b07d4ed6a13b62322b0947565cff33a3fd9"}
Jul 02 21:19:59.180053 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:59.180006    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fd5sb" podStartSLOduration=67.020497721 podStartE2EDuration="1m8.17997427s" totalImagesPullingTime="1.159476549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:19:59.178488569 +0000 UTC m=+67.912121921" watchObservedRunningTime="2026-07-02 21:19:59.17997427 +0000 UTC m=+67.913607577"
Jul 02 21:19:59.305485 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:59.305451    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:19:59.405617 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:19:59.405564    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:00.738484 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:00.738447    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:01.435910 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:01.435873    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:01.549928 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:01.549887    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:03.128738 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:03.128695    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:03.177266 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:03.177236    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mdqbk" event={"ID":"71429679-06c0-480b-8c79-08a2c5682ca1","Type":"ContainerStarted","Data":"67430cac58e9435ecf5bffbe6b79bb44f9c2f01794edd54f91df360c15525401"}
Jul 02 21:20:03.178650 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:03.178598    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pjcpf" event={"ID":"a723a652-5f75-4d27-acc7-5489c2aba607","Type":"ContainerStarted","Data":"382c347597fd5982eb05aa4659ee4d9c6910b115f1aa007c17cc41db50350372"}
Jul 02 21:20:03.178755 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:03.178679    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:20:03.219715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:03.219663    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mdqbk" podStartSLOduration=66.616484256 podStartE2EDuration="1m11.219648478s" totalImagesPullingTime="4.603164222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 21:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:03.218692584 +0000 UTC m=+71.952325894" watchObservedRunningTime="2026-07-02 21:20:03.219648478 +0000 UTC m=+71.953281785"
Jul 02 21:20:03.263917 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:03.263871    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pjcpf" podStartSLOduration=66.955402916 podStartE2EDuration="1m11.26385607s" totalImagesPullingTime="4.308453154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:03.262817002 +0000 UTC m=+71.996450311" watchObservedRunningTime="2026-07-02 21:20:03.26385607 +0000 UTC m=+71.997489376"
Jul 02 21:20:03.368803 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:03.368768    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:03.712008 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:03.711942    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:04.030237 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.030148    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:04.136566 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.136534    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bpxbf"]
Jul 02 21:20:04.170086 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.170059    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.180886 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.180859    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 21:20:04.181090 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.180923    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jmtvw\""
Jul 02 21:20:04.181786 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.181769    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 21:20:04.182587 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.182570    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 21:20:04.182940 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.182923    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 21:20:04.182940 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.182934    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 21:20:04.183733 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.183719    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 21:20:04.198705 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.198665    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:04.283281 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.283205    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-textfile\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.283281 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.283235    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-wtmp\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.283281 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.283255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-tls\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.283522 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.283302    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08e8efc8-577a-42b2-be73-75dad1ba7702-metrics-client-ca\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.283522 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.283356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-accelerators-collector-config\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.283522 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.283420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxbg\" (UniqueName: \"kubernetes.io/projected/08e8efc8-577a-42b2-be73-75dad1ba7702-kube-api-access-swxbg\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.283522 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.283452    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08e8efc8-577a-42b2-be73-75dad1ba7702-sys\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.283522 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.283475    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08e8efc8-577a-42b2-be73-75dad1ba7702-root\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.283751 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.283534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384044 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-textfile\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384044 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384047    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-wtmp\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384269 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-tls\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384269 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384095    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08e8efc8-577a-42b2-be73-75dad1ba7702-metrics-client-ca\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384269 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-accelerators-collector-config\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384269 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxbg\" (UniqueName: \"kubernetes.io/projected/08e8efc8-577a-42b2-be73-75dad1ba7702-kube-api-access-swxbg\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384269 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384191    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08e8efc8-577a-42b2-be73-75dad1ba7702-sys\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384269 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384227    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08e8efc8-577a-42b2-be73-75dad1ba7702-root\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384538 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384262    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-wtmp\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384538 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384538 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384372    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-textfile\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384538 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384375    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08e8efc8-577a-42b2-be73-75dad1ba7702-sys\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384538 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384459    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08e8efc8-577a-42b2-be73-75dad1ba7702-root\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384782 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384763    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08e8efc8-577a-42b2-be73-75dad1ba7702-metrics-client-ca\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.384848 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.384828    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-accelerators-collector-config\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.386424 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.386404    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.386635 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.386615    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08e8efc8-577a-42b2-be73-75dad1ba7702-node-exporter-tls\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.402101 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.402064    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxbg\" (UniqueName: \"kubernetes.io/projected/08e8efc8-577a-42b2-be73-75dad1ba7702-kube-api-access-swxbg\") pod \"node-exporter-bpxbf\" (UID: \"08e8efc8-577a-42b2-be73-75dad1ba7702\") " pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.448010 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.447957    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:04.479383 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.479324    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bpxbf"
Jul 02 21:20:04.521863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:20:04.521832    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e8efc8_577a_42b2_be73_75dad1ba7702.slice/crio-80cb4f7374661aa2ba6410ae4406276f656640b0d56e6d29061f9587ef5a0f38 WatchSource:0}: Error finding container 80cb4f7374661aa2ba6410ae4406276f656640b0d56e6d29061f9587ef5a0f38: Status 404 returned error can't find the container with id 80cb4f7374661aa2ba6410ae4406276f656640b0d56e6d29061f9587ef5a0f38
Jul 02 21:20:04.626882 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.626784    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:04.843131 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.843094    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:04.922775 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:04.922695    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:05.186736 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:05.186634    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpxbf" event={"ID":"08e8efc8-577a-42b2-be73-75dad1ba7702","Type":"ContainerStarted","Data":"80cb4f7374661aa2ba6410ae4406276f656640b0d56e6d29061f9587ef5a0f38"}
Jul 02 21:20:06.191739 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:06.191704    2564 generic.go:356] "Generic (PLEG): container finished" podID="08e8efc8-577a-42b2-be73-75dad1ba7702" containerID="acabdec6769b66a6b6ea07cbb5e48df851535b6d3465a15268162b6f1c646cc0" exitCode=0
Jul 02 21:20:06.192132 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:06.191796    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpxbf" event={"ID":"08e8efc8-577a-42b2-be73-75dad1ba7702","Type":"ContainerDied","Data":"acabdec6769b66a6b6ea07cbb5e48df851535b6d3465a15268162b6f1c646cc0"}
Jul 02 21:20:06.345222 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:06.345191    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:06.436750 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:06.436711    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:06.547037 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:06.546977    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:06.676445 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:06.676408    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:06.945137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:06.945067    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:20:07.059798 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:07.059766    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:07.197839 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:07.197811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpxbf" event={"ID":"08e8efc8-577a-42b2-be73-75dad1ba7702","Type":"ContainerStarted","Data":"726904a9e6c76e057edd6339da838678957775a45c405026d417979f537e57c6"}
Jul 02 21:20:07.198259 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:07.197859    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpxbf" event={"ID":"08e8efc8-577a-42b2-be73-75dad1ba7702","Type":"ContainerStarted","Data":"46f7caa8be5e3e386f8ec90fa406683e8f25ea2e98f0bb9e19581559f61c6af9"}
Jul 02 21:20:07.219003 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:07.218931    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bpxbf" podStartSLOduration=2.199477617 podStartE2EDuration="3.21891756s" totalImagesPullingTime="958.868236ms" totalInitContainerRuntime="60.571707ms" isStatefulPod=true podCreationTimestamp="2026-07-02 21:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:07.218068339 +0000 UTC m=+75.951701645" watchObservedRunningTime="2026-07-02 21:20:07.21891756 +0000 UTC m=+75.952550910"
Jul 02 21:20:07.598746 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:07.598659    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:07.699570 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:07.699539    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:08.093297 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.093244    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:08.210521 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.210483    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:08.425971 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.425891    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"]
Jul 02 21:20:08.428915 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.428900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.431494 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.431471    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 21:20:08.431603 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.431545    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 02 21:20:08.432638 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.432612    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7it3uq5h8tcjg\""
Jul 02 21:20:08.432638 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.432627    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 02 21:20:08.432795 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.432646    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vgd46\""
Jul 02 21:20:08.432795 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.432687    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 02 21:20:08.437948 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.437927    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"]
Jul 02 21:20:08.517983 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.517939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c10ba939-b295-4a3b-a3e0-68a34e5bff76-audit-log\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.517983 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.517987    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c10ba939-b295-4a3b-a3e0-68a34e5bff76-secret-metrics-server-tls\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.518231 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.518080    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c10ba939-b295-4a3b-a3e0-68a34e5bff76-client-ca-bundle\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.518231 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.518141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10ba939-b295-4a3b-a3e0-68a34e5bff76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.518231 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.518192    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c10ba939-b295-4a3b-a3e0-68a34e5bff76-secret-metrics-server-client-certs\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.518231 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.518213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c10ba939-b295-4a3b-a3e0-68a34e5bff76-metrics-server-audit-profiles\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.518371 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.518231    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxwl\" (UniqueName: \"kubernetes.io/projected/c10ba939-b295-4a3b-a3e0-68a34e5bff76-kube-api-access-5wxwl\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.618911 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.618862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10ba939-b295-4a3b-a3e0-68a34e5bff76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.618911 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.618914    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c10ba939-b295-4a3b-a3e0-68a34e5bff76-secret-metrics-server-client-certs\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.619181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.618935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c10ba939-b295-4a3b-a3e0-68a34e5bff76-metrics-server-audit-profiles\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.619181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.618952    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxwl\" (UniqueName: \"kubernetes.io/projected/c10ba939-b295-4a3b-a3e0-68a34e5bff76-kube-api-access-5wxwl\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.619181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.618973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c10ba939-b295-4a3b-a3e0-68a34e5bff76-audit-log\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.619181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.618989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c10ba939-b295-4a3b-a3e0-68a34e5bff76-secret-metrics-server-tls\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.619181 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.619041    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c10ba939-b295-4a3b-a3e0-68a34e5bff76-client-ca-bundle\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.619479 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.619451    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c10ba939-b295-4a3b-a3e0-68a34e5bff76-audit-log\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.619741 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.619704    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10ba939-b295-4a3b-a3e0-68a34e5bff76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.620068 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.620048    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c10ba939-b295-4a3b-a3e0-68a34e5bff76-metrics-server-audit-profiles\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.621522 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.621487    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c10ba939-b295-4a3b-a3e0-68a34e5bff76-secret-metrics-server-client-certs\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.621625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.621597    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c10ba939-b295-4a3b-a3e0-68a34e5bff76-client-ca-bundle\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.621683 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.621600    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c10ba939-b295-4a3b-a3e0-68a34e5bff76-secret-metrics-server-tls\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.627542 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.627521    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxwl\" (UniqueName: \"kubernetes.io/projected/c10ba939-b295-4a3b-a3e0-68a34e5bff76-kube-api-access-5wxwl\") pod \"metrics-server-84b9bd8c9c-qm4kg\" (UID: \"c10ba939-b295-4a3b-a3e0-68a34e5bff76\") " pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.737367 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.737319    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:08.859950 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:08.859923    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"]
Jul 02 21:20:08.862069 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:20:08.862046    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10ba939_b295_4a3b_a3e0_68a34e5bff76.slice/crio-10f4a46c3b9b16ceb0926c74fa1ad5534008378a3917baa43ffc1f423d12baf9 WatchSource:0}: Error finding container 10f4a46c3b9b16ceb0926c74fa1ad5534008378a3917baa43ffc1f423d12baf9: Status 404 returned error can't find the container with id 10f4a46c3b9b16ceb0926c74fa1ad5534008378a3917baa43ffc1f423d12baf9
Jul 02 21:20:09.121355 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:09.121279    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f445984c5-q9n6l"
Jul 02 21:20:09.205749 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:09.205717    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg" event={"ID":"c10ba939-b295-4a3b-a3e0-68a34e5bff76","Type":"ContainerStarted","Data":"10f4a46c3b9b16ceb0926c74fa1ad5534008378a3917baa43ffc1f423d12baf9"}
Jul 02 21:20:09.286776 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:09.286746    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:09.376772 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:09.376686    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:09.919558 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:09.919498    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:11.213665 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:11.213628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg" event={"ID":"c10ba939-b295-4a3b-a3e0-68a34e5bff76","Type":"ContainerStarted","Data":"f1831cb755ff5aff5daeace806784116f307dbc08053436b97e3369ca9e25a2a"}
Jul 02 21:20:11.235260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:11.235215    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg" podStartSLOduration=1.8596687140000001 podStartE2EDuration="3.235198302s" totalImagesPullingTime="1.375529588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:11.234876791 +0000 UTC m=+79.968510128" watchObservedRunningTime="2026-07-02 21:20:11.235198302 +0000 UTC m=+79.968831610"
Jul 02 21:20:11.958715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:11.958652    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn" podUID="7314ec50-bab2-4b39-b051-cdd84c7b9d8e" containerName="registry" containerID="cri-o://d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85" gracePeriod=30
Jul 02 21:20:12.192861 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.192837    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:20:12.218664 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.218473    2564 generic.go:356] "Generic (PLEG): container finished" podID="7314ec50-bab2-4b39-b051-cdd84c7b9d8e" containerID="d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85" exitCode=0
Jul 02 21:20:12.218664 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.218505    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn" event={"ID":"7314ec50-bab2-4b39-b051-cdd84c7b9d8e","Type":"ContainerDied","Data":"d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85"}
Jul 02 21:20:12.218664 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.218556    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn"
Jul 02 21:20:12.218664 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.218566    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8f494dc48-hzdrn" event={"ID":"7314ec50-bab2-4b39-b051-cdd84c7b9d8e","Type":"ContainerDied","Data":"7cb307b5f61ca9a9bf3651cdfa866fa4570d907b574040aa4309f98580735b4b"}
Jul 02 21:20:12.219336 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.219315    2564 scope.go:117] "RemoveContainer" containerID="d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85"
Jul 02 21:20:12.227197 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.227177    2564 scope.go:117] "RemoveContainer" containerID="d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85"
Jul 02 21:20:12.227456 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:20:12.227440    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85\": container with ID starting with d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85 not found: ID does not exist" containerID="d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85"
Jul 02 21:20:12.227525 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.227468    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85"} err="failed to get container status \"d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85\": rpc error: code = NotFound desc = could not find container \"d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85\": container with ID starting with d476009c1eca43c0042e882a76ce0f697e8697766d88a0eb8ea4b7eeba0f1f85 not found: ID does not exist"
Jul 02 21:20:12.350060 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.350029    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-ca-trust-extracted\") pod \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") "
Jul 02 21:20:12.350060 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.350071    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-image-registry-private-configuration\") pod \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") "
Jul 02 21:20:12.350328 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.350098    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-installation-pull-secrets\") pod \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") "
Jul 02 21:20:12.350328 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.350118    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-bound-sa-token\") pod \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") "
Jul 02 21:20:12.350328 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.350144    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-certificates\") pod \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") "
Jul 02 21:20:12.350328 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.350175    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls\") pod \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") "
Jul 02 21:20:12.350328 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.350218    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgjps\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-kube-api-access-xgjps\") pod \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") "
Jul 02 21:20:12.350577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.350346    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-trusted-ca\") pod \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\" (UID: \"7314ec50-bab2-4b39-b051-cdd84c7b9d8e\") "
Jul 02 21:20:12.351272 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.350706    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7314ec50-bab2-4b39-b051-cdd84c7b9d8e" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:20:12.351272 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.351190    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7314ec50-bab2-4b39-b051-cdd84c7b9d8e" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:20:12.352863 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.352837    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7314ec50-bab2-4b39-b051-cdd84c7b9d8e" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:20:12.352963 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.352950    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7314ec50-bab2-4b39-b051-cdd84c7b9d8e" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:20:12.353108 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.353073    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7314ec50-bab2-4b39-b051-cdd84c7b9d8e" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:20:12.353179 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.353088    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7314ec50-bab2-4b39-b051-cdd84c7b9d8e" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:20:12.353179 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.353142    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-kube-api-access-xgjps" (OuterVolumeSpecName: "kube-api-access-xgjps") pod "7314ec50-bab2-4b39-b051-cdd84c7b9d8e" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e"). InnerVolumeSpecName "kube-api-access-xgjps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:20:12.359742 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.359717    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7314ec50-bab2-4b39-b051-cdd84c7b9d8e" (UID: "7314ec50-bab2-4b39-b051-cdd84c7b9d8e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:20:12.451491 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.451447    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-trusted-ca\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:20:12.451491 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.451490    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-ca-trust-extracted\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:20:12.451491 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.451503    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-image-registry-private-configuration\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:20:12.451666 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.451513    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-installation-pull-secrets\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:20:12.451666 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.451523    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-bound-sa-token\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:20:12.451666 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.451531    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-certificates\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:20:12.451666 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.451539    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-registry-tls\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:20:12.451666 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.451549    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgjps\" (UniqueName: \"kubernetes.io/projected/7314ec50-bab2-4b39-b051-cdd84c7b9d8e-kube-api-access-xgjps\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:20:12.553158 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.553125    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8f494dc48-hzdrn"]
Jul 02 21:20:12.563023 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:12.562976    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8f494dc48-hzdrn"]
Jul 02 21:20:13.833523 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:13.833416    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7314ec50-bab2-4b39-b051-cdd84c7b9d8e" path="/var/lib/kubelet/pods/7314ec50-bab2-4b39-b051-cdd84c7b9d8e/volumes"
Jul 02 21:20:18.719520 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.719483    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:18.824619 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.824585    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-z2mms"]
Jul 02 21:20:18.824817 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.824806    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7314ec50-bab2-4b39-b051-cdd84c7b9d8e" containerName="registry"
Jul 02 21:20:18.824864 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.824819    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7314ec50-bab2-4b39-b051-cdd84c7b9d8e" containerName="registry"
Jul 02 21:20:18.824896 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.824885    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7314ec50-bab2-4b39-b051-cdd84c7b9d8e" containerName="registry"
Jul 02 21:20:18.830556 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.830525    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:18.845784 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.845756    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-z2mms"]
Jul 02 21:20:18.845923 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.845905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-z2mms"
Jul 02 21:20:18.848923 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.848901    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 21:20:18.849179 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.849155    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 21:20:18.849276 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.849214    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xmmlq\""
Jul 02 21:20:18.922648 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.922615    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:18.997214 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:18.997137    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g2tp\" (UniqueName: \"kubernetes.io/projected/15e84cd5-bccd-4749-b3d3-2f5d805ab6e6-kube-api-access-4g2tp\") pod \"downloads-685bfb5f64-z2mms\" (UID: \"15e84cd5-bccd-4749-b3d3-2f5d805ab6e6\") " pod="openshift-console/downloads-685bfb5f64-z2mms"
Jul 02 21:20:19.005340 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:19.005294    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:19.098232 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:19.098179    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g2tp\" (UniqueName: \"kubernetes.io/projected/15e84cd5-bccd-4749-b3d3-2f5d805ab6e6-kube-api-access-4g2tp\") pod \"downloads-685bfb5f64-z2mms\" (UID: \"15e84cd5-bccd-4749-b3d3-2f5d805ab6e6\") " pod="openshift-console/downloads-685bfb5f64-z2mms"
Jul 02 21:20:19.108891 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:19.108861    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g2tp\" (UniqueName: \"kubernetes.io/projected/15e84cd5-bccd-4749-b3d3-2f5d805ab6e6-kube-api-access-4g2tp\") pod \"downloads-685bfb5f64-z2mms\" (UID: \"15e84cd5-bccd-4749-b3d3-2f5d805ab6e6\") " pod="openshift-console/downloads-685bfb5f64-z2mms"
Jul 02 21:20:19.135740 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:19.135709    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:19.155565 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:19.155545    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-z2mms"
Jul 02 21:20:19.279590 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:19.279568    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-z2mms"]
Jul 02 21:20:19.281476 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:20:19.281449    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e84cd5_bccd_4749_b3d3_2f5d805ab6e6.slice/crio-c1ca35df26fa3da7fe2104ddec29719156bc4f2e0016017f744263e07c393eb2 WatchSource:0}: Error finding container c1ca35df26fa3da7fe2104ddec29719156bc4f2e0016017f744263e07c393eb2: Status 404 returned error can't find the container with id c1ca35df26fa3da7fe2104ddec29719156bc4f2e0016017f744263e07c393eb2
Jul 02 21:20:19.299136 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:19.299109    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:19.513393 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:19.513357    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:19.613440 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:19.613359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:19.714821 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:19.714785    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:19.792106 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:19.792068    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:20.250882 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:20.250842    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-z2mms" event={"ID":"15e84cd5-bccd-4749-b3d3-2f5d805ab6e6","Type":"ContainerStarted","Data":"c1ca35df26fa3da7fe2104ddec29719156bc4f2e0016017f744263e07c393eb2"}
Jul 02 21:20:20.653067 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:20.652957    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:20.878572 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:20.878457    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:21.079103 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:21.079068    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:21.384222 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:21.384129    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:21.557490 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:21.557448    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:21.656448 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:21.656353    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:22.138905 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:22.138853    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:22.207847 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:22.207810    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:28.642967 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.642928    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:28.738284 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.738234    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:28.738455 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.738326    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:28.743601 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.743567    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:28.912756 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.912692    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7469cc7b8-8kqj4"]
Jul 02 21:20:28.916124 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.916102    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:28.919762 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.919742    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 21:20:28.919866 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.919763    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 21:20:28.921008 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.920967    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 21:20:28.921008 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.920969    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 21:20:28.921168 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.921040    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 21:20:28.921168 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.921064    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hg6kk\""
Jul 02 21:20:28.928023 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:28.927987    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7469cc7b8-8kqj4"]
Jul 02 21:20:29.078965 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.078932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-serving-cert\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.079159 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.078980    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-service-ca\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.079159 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.079015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-oauth-config\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.079159 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.079046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf8q9\" (UniqueName: \"kubernetes.io/projected/69825ffd-580f-4f4f-bbb6-47740690c0b9-kube-api-access-sf8q9\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.079159 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.079076    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-config\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.079159 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.079101    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-oauth-serving-cert\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.180133 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.180052    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-oauth-config\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.180133 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.180100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf8q9\" (UniqueName: \"kubernetes.io/projected/69825ffd-580f-4f4f-bbb6-47740690c0b9-kube-api-access-sf8q9\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.180349 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.180141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-config\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.180349 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.180167    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-oauth-serving-cert\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.180349 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.180230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-serving-cert\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.180349 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.180277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-service-ca\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.181007 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.180940    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-config\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.181145 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.181034    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-oauth-serving-cert\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.181145 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.181059    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-service-ca\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.182932 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.182905    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-oauth-config\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.183046 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.183037    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-serving-cert\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.193733 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.193712    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf8q9\" (UniqueName: \"kubernetes.io/projected/69825ffd-580f-4f4f-bbb6-47740690c0b9-kube-api-access-sf8q9\") pod \"console-7469cc7b8-8kqj4\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") " pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.227537 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.227500    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:20:29.612394 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:29.612358    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:32.164720 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:32.164629    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:32.266881 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:32.266838    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:32.740901 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:32.740864    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:32.840503 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:32.840459    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:33.607095 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:33.607057    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:33.676540 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:33.676505    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:34.184255 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:34.184226    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pjcpf"
Jul 02 21:20:34.344525 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:34.344491    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:34.448505 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:34.448475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:35.881339 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:35.881305    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7469cc7b8-8kqj4"]
Jul 02 21:20:35.884110 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:20:35.884082    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69825ffd_580f_4f4f_bbb6_47740690c0b9.slice/crio-0e047e353645705c218058ba0990d1f7cd87b1f29a99e2f77a2b9fe6c8298e15 WatchSource:0}: Error finding container 0e047e353645705c218058ba0990d1f7cd87b1f29a99e2f77a2b9fe6c8298e15: Status 404 returned error can't find the container with id 0e047e353645705c218058ba0990d1f7cd87b1f29a99e2f77a2b9fe6c8298e15
Jul 02 21:20:36.188472 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.188443    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:36.299473 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.299441    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:36.317967 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.317906    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-z2mms" event={"ID":"15e84cd5-bccd-4749-b3d3-2f5d805ab6e6","Type":"ContainerStarted","Data":"0e95c2a5ec36a5a297c3f0a5375440bb28c339a77857e11676b5506136b94455"}
Jul 02 21:20:36.318593 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.318186    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-z2mms"
Jul 02 21:20:36.319620 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.319596    2564 patch_prober.go:28] interesting pod/downloads-685bfb5f64-z2mms container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" start-of-body=
Jul 02 21:20:36.319712 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.319643    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-z2mms" podUID="15e84cd5-bccd-4749-b3d3-2f5d805ab6e6" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused"
Jul 02 21:20:36.319806 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.319768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7469cc7b8-8kqj4" event={"ID":"69825ffd-580f-4f4f-bbb6-47740690c0b9","Type":"ContainerStarted","Data":"0e047e353645705c218058ba0990d1f7cd87b1f29a99e2f77a2b9fe6c8298e15"}
Jul 02 21:20:36.754532 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.754417    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-z2mms" podStartSLOduration=1.896485918 podStartE2EDuration="18.754393917s" totalImagesPullingTime="16.857907999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:36.347874242 +0000 UTC m=+105.081507562" watchObservedRunningTime="2026-07-02 21:20:36.754393917 +0000 UTC m=+105.488027228"
Jul 02 21:20:36.754532 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.754583    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-678dcd4695-tdbkn"]
Jul 02 21:20:36.778733 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.778654    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.787687 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.787497    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678dcd4695-tdbkn"]
Jul 02 21:20:36.796041 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.796009    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 21:20:36.839946 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.839915    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-trusted-ca-bundle\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.840109 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.839954    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-config\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.840109 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.840009    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-service-ca\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.840202 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.840106    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-oauth-config\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.840246 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.840204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-oauth-serving-cert\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.840246 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.840237    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-serving-cert\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.840339 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.840255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnmxb\" (UniqueName: \"kubernetes.io/projected/1c90d527-a551-43e4-a4e5-d3bf81532c46-kube-api-access-qnmxb\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.941448 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.941407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-oauth-config\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.941895 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.941477    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-oauth-serving-cert\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.941895 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.941503    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-serving-cert\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.941895 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.941532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnmxb\" (UniqueName: \"kubernetes.io/projected/1c90d527-a551-43e4-a4e5-d3bf81532c46-kube-api-access-qnmxb\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.941895 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.941570    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-trusted-ca-bundle\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.941895 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.941603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-config\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.941895 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.941646    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-service-ca\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.942448 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.942425    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-service-ca\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.943383 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.943336    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-oauth-serving-cert\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.943726 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.943699    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-config\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.944552 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.944526    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-trusted-ca-bundle\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.948722 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.948408    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-serving-cert\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.948722 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.948681    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-oauth-config\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:36.957977 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:36.957929    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnmxb\" (UniqueName: \"kubernetes.io/projected/1c90d527-a551-43e4-a4e5-d3bf81532c46-kube-api-access-qnmxb\") pod \"console-678dcd4695-tdbkn\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") " pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:37.057922 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:37.057836    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84b9bd8c9c-qm4kg_c10ba939-b295-4a3b-a3e0-68a34e5bff76/metrics-server/0.log"
Jul 02 21:20:37.095063 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:37.095028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:37.298393 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:37.298064    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678dcd4695-tdbkn"]
Jul 02 21:20:37.299863 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:20:37.299832    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c90d527_a551_43e4_a4e5_d3bf81532c46.slice/crio-c518755a88974fbfd6c8958cdebaf5886494d52c7ac2df603f72adf6c7dc8ed9 WatchSource:0}: Error finding container c518755a88974fbfd6c8958cdebaf5886494d52c7ac2df603f72adf6c7dc8ed9: Status 404 returned error can't find the container with id c518755a88974fbfd6c8958cdebaf5886494d52c7ac2df603f72adf6c7dc8ed9
Jul 02 21:20:37.328833 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:37.328726    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678dcd4695-tdbkn" event={"ID":"1c90d527-a551-43e4-a4e5-d3bf81532c46","Type":"ContainerStarted","Data":"c518755a88974fbfd6c8958cdebaf5886494d52c7ac2df603f72adf6c7dc8ed9"}
Jul 02 21:20:37.342336 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:37.342309    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-z2mms"
Jul 02 21:20:38.058315 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:38.058276    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpxbf_08e8efc8-577a-42b2-be73-75dad1ba7702/init-textfile/0.log"
Jul 02 21:20:38.260407 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:38.260286    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpxbf_08e8efc8-577a-42b2-be73-75dad1ba7702/node-exporter/0.log"
Jul 02 21:20:38.458441 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:38.458404    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpxbf_08e8efc8-577a-42b2-be73-75dad1ba7702/kube-rbac-proxy/0.log"
Jul 02 21:20:40.343549 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:40.343507    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7469cc7b8-8kqj4" event={"ID":"69825ffd-580f-4f4f-bbb6-47740690c0b9","Type":"ContainerStarted","Data":"55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120"}
Jul 02 21:20:40.345309 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:40.345280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678dcd4695-tdbkn" event={"ID":"1c90d527-a551-43e4-a4e5-d3bf81532c46","Type":"ContainerStarted","Data":"58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174"}
Jul 02 21:20:40.362965 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:40.362919    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7469cc7b8-8kqj4" podStartSLOduration=8.659194407 podStartE2EDuration="12.362900574s" totalImagesPullingTime="3.703706167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:40.361759949 +0000 UTC m=+109.095393275" watchObservedRunningTime="2026-07-02 21:20:40.362900574 +0000 UTC m=+109.096533883"
Jul 02 21:20:40.400790 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:40.400733    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-678dcd4695-tdbkn" podStartSLOduration=2.113301315 podStartE2EDuration="4.400717009s" totalImagesPullingTime="2.287415694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:20:40.399092507 +0000 UTC m=+109.132725815" watchObservedRunningTime="2026-07-02 21:20:40.400717009 +0000 UTC m=+109.134350354"
Jul 02 21:20:40.736311 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:40.736222    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:40.836502 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:40.836465    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:41.657085 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:41.657047    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-rtdsl_a7dd804b-09c4-4d04-b3c4-ec3f1fdaef06/prometheus-operator-admission-webhook/0.log"
Jul 02 21:20:43.659085 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:43.659051    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-shwvz_287c4ada-4462-484f-9832-b617173a665f/networking-console-plugin/0.log"
Jul 02 21:20:44.257321 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:44.257287    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7469cc7b8-8kqj4_69825ffd-580f-4f4f-bbb6-47740690c0b9/console/0.log"
Jul 02 21:20:44.459465 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:44.459429    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-z2mms_15e84cd5-bccd-4749-b3d3-2f5d805ab6e6/download-server/0.log"
Jul 02 21:20:44.916083 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:44.916051    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:45.017498 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:45.017464    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:47.095950 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:47.095914    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:47.096492 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:47.095976    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:47.101372 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:47.101345    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:47.382384 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:47.382301    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:20:47.431241 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:47.431208    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7469cc7b8-8kqj4"]
Jul 02 21:20:47.585661 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:47.585628    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:48.197291 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:48.197254    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:48.273535 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:48.273502    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:48.606532 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:48.606445    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:48.671042 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:48.671010    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:20:48.743078 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:48.743053    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:48.746811 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:48.746791    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84b9bd8c9c-qm4kg"
Jul 02 21:20:49.228183 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:20:49.228144    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:21:03.609066 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:03.609027    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:03.683966 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:03.683924    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:10.500528 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:10.500489    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:12.458266 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.458162    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7469cc7b8-8kqj4" podUID="69825ffd-580f-4f4f-bbb6-47740690c0b9" containerName="console" containerID="cri-o://55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120" gracePeriod=15
Jul 02 21:21:12.717553 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.717492    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:21:12.835852 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.835822    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-oauth-config\") pod \"69825ffd-580f-4f4f-bbb6-47740690c0b9\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") "
Jul 02 21:21:12.836049 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.835866    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-oauth-serving-cert\") pod \"69825ffd-580f-4f4f-bbb6-47740690c0b9\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") "
Jul 02 21:21:12.836049 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.835910    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-config\") pod \"69825ffd-580f-4f4f-bbb6-47740690c0b9\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") "
Jul 02 21:21:12.836183 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.836043    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-serving-cert\") pod \"69825ffd-580f-4f4f-bbb6-47740690c0b9\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") "
Jul 02 21:21:12.836183 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.836098    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf8q9\" (UniqueName: \"kubernetes.io/projected/69825ffd-580f-4f4f-bbb6-47740690c0b9-kube-api-access-sf8q9\") pod \"69825ffd-580f-4f4f-bbb6-47740690c0b9\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") "
Jul 02 21:21:12.836183 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.836140    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-service-ca\") pod \"69825ffd-580f-4f4f-bbb6-47740690c0b9\" (UID: \"69825ffd-580f-4f4f-bbb6-47740690c0b9\") "
Jul 02 21:21:12.836326 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.836290    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-config" (OuterVolumeSpecName: "console-config") pod "69825ffd-580f-4f4f-bbb6-47740690c0b9" (UID: "69825ffd-580f-4f4f-bbb6-47740690c0b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:12.836326 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.836305    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "69825ffd-580f-4f4f-bbb6-47740690c0b9" (UID: "69825ffd-580f-4f4f-bbb6-47740690c0b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:12.836522 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.836502    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-oauth-serving-cert\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:21:12.836593 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.836528    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:21:12.836593 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.836556    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-service-ca" (OuterVolumeSpecName: "service-ca") pod "69825ffd-580f-4f4f-bbb6-47740690c0b9" (UID: "69825ffd-580f-4f4f-bbb6-47740690c0b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:21:12.838208 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.838174    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69825ffd-580f-4f4f-bbb6-47740690c0b9-kube-api-access-sf8q9" (OuterVolumeSpecName: "kube-api-access-sf8q9") pod "69825ffd-580f-4f4f-bbb6-47740690c0b9" (UID: "69825ffd-580f-4f4f-bbb6-47740690c0b9"). InnerVolumeSpecName "kube-api-access-sf8q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:21:12.838325 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.838225    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "69825ffd-580f-4f4f-bbb6-47740690c0b9" (UID: "69825ffd-580f-4f4f-bbb6-47740690c0b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:12.838325 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.838249    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "69825ffd-580f-4f4f-bbb6-47740690c0b9" (UID: "69825ffd-580f-4f4f-bbb6-47740690c0b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:21:12.936938 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.936906    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-serving-cert\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:21:12.936938 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.936931    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf8q9\" (UniqueName: \"kubernetes.io/projected/69825ffd-580f-4f4f-bbb6-47740690c0b9-kube-api-access-sf8q9\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:21:12.936938 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.936942    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/69825ffd-580f-4f4f-bbb6-47740690c0b9-service-ca\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:21:12.937162 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:12.936951    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/69825ffd-580f-4f4f-bbb6-47740690c0b9-console-oauth-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:21:13.463782 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:13.463748    2564 generic.go:356] "Generic (PLEG): container finished" podID="69825ffd-580f-4f4f-bbb6-47740690c0b9" containerID="55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120" exitCode=0
Jul 02 21:21:13.464234 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:13.463791    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7469cc7b8-8kqj4"
Jul 02 21:21:13.464234 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:13.463803    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7469cc7b8-8kqj4" event={"ID":"69825ffd-580f-4f4f-bbb6-47740690c0b9","Type":"ContainerDied","Data":"55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120"}
Jul 02 21:21:13.464234 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:13.463846    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7469cc7b8-8kqj4" event={"ID":"69825ffd-580f-4f4f-bbb6-47740690c0b9","Type":"ContainerDied","Data":"0e047e353645705c218058ba0990d1f7cd87b1f29a99e2f77a2b9fe6c8298e15"}
Jul 02 21:21:13.464234 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:13.463865    2564 scope.go:117] "RemoveContainer" containerID="55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120"
Jul 02 21:21:13.472682 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:13.472662    2564 scope.go:117] "RemoveContainer" containerID="55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120"
Jul 02 21:21:13.472940 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:21:13.472920    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120\": container with ID starting with 55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120 not found: ID does not exist" containerID="55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120"
Jul 02 21:21:13.473024 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:13.472947    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120"} err="failed to get container status \"55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120\": rpc error: code = NotFound desc = could not find container \"55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120\": container with ID starting with 55a9e2fda7e8ad69b64266257adf725d340414a9708963b7b6a9472ce93bd120 not found: ID does not exist"
Jul 02 21:21:13.486370 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:13.486349    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7469cc7b8-8kqj4"]
Jul 02 21:21:13.490066 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:13.490046    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7469cc7b8-8kqj4"]
Jul 02 21:21:13.832918 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:13.832839    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69825ffd-580f-4f4f-bbb6-47740690c0b9" path="/var/lib/kubelet/pods/69825ffd-580f-4f4f-bbb6-47740690c0b9/volumes"
Jul 02 21:21:14.200357 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:14.200322    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:14.266923 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:14.266887    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:19.715246 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:19.715201    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:19.815069 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:19.815031    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:31.612233 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:31.612194    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:31.667430 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:31.667391    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:48.596347 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:48.596233    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:48.696918 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:48.696886    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:50.245625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:50.245589    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:21:50.330693 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:21:50.330653    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:05.231913 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:05.231878    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:06.281297 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.281260    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b674bdc77-xn64j"]
Jul 02 21:22:06.281760 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.281507    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69825ffd-580f-4f4f-bbb6-47740690c0b9" containerName="console"
Jul 02 21:22:06.281760 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.281519    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="69825ffd-580f-4f4f-bbb6-47740690c0b9" containerName="console"
Jul 02 21:22:06.281760 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.281562    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="69825ffd-580f-4f4f-bbb6-47740690c0b9" containerName="console"
Jul 02 21:22:06.285458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.285439    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.289450 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.289421    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b674bdc77-xn64j"]
Jul 02 21:22:06.315957 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.315930    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-trusted-ca-bundle\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.316113 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.315965    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-oauth-config\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.316113 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.316049    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-serving-cert\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.316203 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.316105    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-config\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.316203 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.316149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-oauth-serving-cert\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.316203 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.316170    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-service-ca\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.316203 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.316191    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-775r6\" (UniqueName: \"kubernetes.io/projected/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-kube-api-access-775r6\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.341957 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.341920    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:06.416583 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.416550    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-trusted-ca-bundle\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.416583 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.416586    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-oauth-config\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.416778 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.416618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-serving-cert\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.416778 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.416729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-config\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.416875 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.416778    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-oauth-serving-cert\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.416875 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.416799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-service-ca\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.416974 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.416909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-775r6\" (UniqueName: \"kubernetes.io/projected/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-kube-api-access-775r6\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.417484 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.417455    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-config\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.417593 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.417500    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-service-ca\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.417593 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.417538    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-trusted-ca-bundle\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.417593 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.417545    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-oauth-serving-cert\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.419618 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.419595    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-serving-cert\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.419737 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.419718    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-oauth-config\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.425532 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.425514    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-775r6\" (UniqueName: \"kubernetes.io/projected/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-kube-api-access-775r6\") pod \"console-b674bdc77-xn64j\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") " pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.436460 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.436433    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:06.594864 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.594765    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:06.725858 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:06.725828    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b674bdc77-xn64j"]
Jul 02 21:22:06.728929 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:22:06.728897    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d7fdf5_3fee_40fb_ac9a_742932cf55a2.slice/crio-1585fa177daf53d83b66beecde734a271a9787f5a9bf8d2f67ef46b1b45a15b3 WatchSource:0}: Error finding container 1585fa177daf53d83b66beecde734a271a9787f5a9bf8d2f67ef46b1b45a15b3: Status 404 returned error can't find the container with id 1585fa177daf53d83b66beecde734a271a9787f5a9bf8d2f67ef46b1b45a15b3
Jul 02 21:22:07.643325 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:07.643286    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b674bdc77-xn64j" event={"ID":"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2","Type":"ContainerStarted","Data":"d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452"}
Jul 02 21:22:07.643761 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:07.643338    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b674bdc77-xn64j" event={"ID":"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2","Type":"ContainerStarted","Data":"1585fa177daf53d83b66beecde734a271a9787f5a9bf8d2f67ef46b1b45a15b3"}
Jul 02 21:22:07.661776 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:07.661724    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b674bdc77-xn64j" podStartSLOduration=1.6617058519999999 podStartE2EDuration="1.661705852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:22:07.660448148 +0000 UTC m=+196.394081481" watchObservedRunningTime="2026-07-02 21:22:07.661705852 +0000 UTC m=+196.395339160"
Jul 02 21:22:16.594979 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:16.594935    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:16.595459 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:16.595023    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:16.599739 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:16.599715    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:16.677926 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:16.677899    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:22:16.724955 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:16.724924    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678dcd4695-tdbkn"]
Jul 02 21:22:16.834648 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:16.834615    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:16.935478 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:16.935397    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:16.994309 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:16.994282    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:18.160145 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.160110    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h"]
Jul 02 21:22:18.163140 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.163122    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h"
Jul 02 21:22:18.165653 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.165629    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 21:22:18.166890 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.166869    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 02 21:22:18.167015 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.166896    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 21:22:18.167015 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.166941    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 21:22:18.172473 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.172448    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h"]
Jul 02 21:22:18.190832 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.190798    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:18.206896 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.206858    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6cd5e2d9-505f-4610-b230-37aad7303140-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b47fdd954-7ng9h\" (UID: \"6cd5e2d9-505f-4610-b230-37aad7303140\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h"
Jul 02 21:22:18.207049 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.206929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbwm\" (UniqueName: \"kubernetes.io/projected/6cd5e2d9-505f-4610-b230-37aad7303140-kube-api-access-vkbwm\") pod \"managed-serviceaccount-addon-agent-b47fdd954-7ng9h\" (UID: \"6cd5e2d9-505f-4610-b230-37aad7303140\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h"
Jul 02 21:22:18.273350 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.273314    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:18.308007 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.307966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbwm\" (UniqueName: \"kubernetes.io/projected/6cd5e2d9-505f-4610-b230-37aad7303140-kube-api-access-vkbwm\") pod \"managed-serviceaccount-addon-agent-b47fdd954-7ng9h\" (UID: \"6cd5e2d9-505f-4610-b230-37aad7303140\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h"
Jul 02 21:22:18.308176 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.308032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6cd5e2d9-505f-4610-b230-37aad7303140-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b47fdd954-7ng9h\" (UID: \"6cd5e2d9-505f-4610-b230-37aad7303140\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h"
Jul 02 21:22:18.310534 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.310500    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6cd5e2d9-505f-4610-b230-37aad7303140-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b47fdd954-7ng9h\" (UID: \"6cd5e2d9-505f-4610-b230-37aad7303140\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h"
Jul 02 21:22:18.318394 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.318368    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbwm\" (UniqueName: \"kubernetes.io/projected/6cd5e2d9-505f-4610-b230-37aad7303140-kube-api-access-vkbwm\") pod \"managed-serviceaccount-addon-agent-b47fdd954-7ng9h\" (UID: \"6cd5e2d9-505f-4610-b230-37aad7303140\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h"
Jul 02 21:22:18.334606 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.334568    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:22:18.487884 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.487853    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h"
Jul 02 21:22:18.621553 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.621476    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h"]
Jul 02 21:22:18.624367 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:22:18.624335    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cd5e2d9_505f_4610_b230_37aad7303140.slice/crio-1e21d7d9bf962fa671cbec16636b5e6d55055274c7da3c895a1df079dbf83f24 WatchSource:0}: Error finding container 1e21d7d9bf962fa671cbec16636b5e6d55055274c7da3c895a1df079dbf83f24: Status 404 returned error can't find the container with id 1e21d7d9bf962fa671cbec16636b5e6d55055274c7da3c895a1df079dbf83f24
Jul 02 21:22:18.682006 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:18.681970    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h" event={"ID":"6cd5e2d9-505f-4610-b230-37aad7303140","Type":"ContainerStarted","Data":"1e21d7d9bf962fa671cbec16636b5e6d55055274c7da3c895a1df079dbf83f24"}
Jul 02 21:22:22.699654 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:22.699621    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h" event={"ID":"6cd5e2d9-505f-4610-b230-37aad7303140","Type":"ContainerStarted","Data":"b8dde050393cfc0299cf7608c99b964e5ad59dde8070174f1a843a4cd5875213"}
Jul 02 21:22:22.715660 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:22.715603    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b47fdd954-7ng9h" podStartSLOduration=1.129743063 podStartE2EDuration="4.715587553s" totalImagesPullingTime="3.58584449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:22:22.714600777 +0000 UTC m=+211.448234088" watchObservedRunningTime="2026-07-02 21:22:22.715587553 +0000 UTC m=+211.449220860"
Jul 02 21:22:41.744655 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:41.744620    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-678dcd4695-tdbkn" podUID="1c90d527-a551-43e4-a4e5-d3bf81532c46" containerName="console" containerID="cri-o://58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174" gracePeriod=15
Jul 02 21:22:41.980823 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:41.980797    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:22:42.100541 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.100451    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-service-ca\") pod \"1c90d527-a551-43e4-a4e5-d3bf81532c46\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") "
Jul 02 21:22:42.100541 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.100496    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnmxb\" (UniqueName: \"kubernetes.io/projected/1c90d527-a551-43e4-a4e5-d3bf81532c46-kube-api-access-qnmxb\") pod \"1c90d527-a551-43e4-a4e5-d3bf81532c46\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") "
Jul 02 21:22:42.100541 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.100535    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-serving-cert\") pod \"1c90d527-a551-43e4-a4e5-d3bf81532c46\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") "
Jul 02 21:22:42.100816 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.100564    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-oauth-serving-cert\") pod \"1c90d527-a551-43e4-a4e5-d3bf81532c46\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") "
Jul 02 21:22:42.100816 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.100599    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-config\") pod \"1c90d527-a551-43e4-a4e5-d3bf81532c46\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") "
Jul 02 21:22:42.100816 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.100630    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-oauth-config\") pod \"1c90d527-a551-43e4-a4e5-d3bf81532c46\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") "
Jul 02 21:22:42.100816 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.100679    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-trusted-ca-bundle\") pod \"1c90d527-a551-43e4-a4e5-d3bf81532c46\" (UID: \"1c90d527-a551-43e4-a4e5-d3bf81532c46\") "
Jul 02 21:22:42.101032 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.100963    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-service-ca" (OuterVolumeSpecName: "service-ca") pod "1c90d527-a551-43e4-a4e5-d3bf81532c46" (UID: "1c90d527-a551-43e4-a4e5-d3bf81532c46"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:22:42.101132 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.101106    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1c90d527-a551-43e4-a4e5-d3bf81532c46" (UID: "1c90d527-a551-43e4-a4e5-d3bf81532c46"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:22:42.101180 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.101134    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-config" (OuterVolumeSpecName: "console-config") pod "1c90d527-a551-43e4-a4e5-d3bf81532c46" (UID: "1c90d527-a551-43e4-a4e5-d3bf81532c46"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:22:42.101233 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.101203    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1c90d527-a551-43e4-a4e5-d3bf81532c46" (UID: "1c90d527-a551-43e4-a4e5-d3bf81532c46"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:22:42.102770 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.102746    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c90d527-a551-43e4-a4e5-d3bf81532c46-kube-api-access-qnmxb" (OuterVolumeSpecName: "kube-api-access-qnmxb") pod "1c90d527-a551-43e4-a4e5-d3bf81532c46" (UID: "1c90d527-a551-43e4-a4e5-d3bf81532c46"). InnerVolumeSpecName "kube-api-access-qnmxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:22:42.102879 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.102791    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1c90d527-a551-43e4-a4e5-d3bf81532c46" (UID: "1c90d527-a551-43e4-a4e5-d3bf81532c46"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:22:42.102879 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.102812    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1c90d527-a551-43e4-a4e5-d3bf81532c46" (UID: "1c90d527-a551-43e4-a4e5-d3bf81532c46"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:22:42.201546 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.201516    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:22:42.201546 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.201543    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-oauth-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:22:42.201546 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.201554    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-trusted-ca-bundle\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:22:42.201755 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.201563    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-service-ca\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:22:42.201755 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.201572    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnmxb\" (UniqueName: \"kubernetes.io/projected/1c90d527-a551-43e4-a4e5-d3bf81532c46-kube-api-access-qnmxb\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:22:42.201755 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.201581    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c90d527-a551-43e4-a4e5-d3bf81532c46-console-serving-cert\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:22:42.201755 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.201589    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c90d527-a551-43e4-a4e5-d3bf81532c46-oauth-serving-cert\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:22:42.772137 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.772105    2564 generic.go:356] "Generic (PLEG): container finished" podID="1c90d527-a551-43e4-a4e5-d3bf81532c46" containerID="58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174" exitCode=0
Jul 02 21:22:42.772525 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.772149    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678dcd4695-tdbkn"
Jul 02 21:22:42.772525 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.772155    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678dcd4695-tdbkn" event={"ID":"1c90d527-a551-43e4-a4e5-d3bf81532c46","Type":"ContainerDied","Data":"58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174"}
Jul 02 21:22:42.772525 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.772194    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678dcd4695-tdbkn" event={"ID":"1c90d527-a551-43e4-a4e5-d3bf81532c46","Type":"ContainerDied","Data":"c518755a88974fbfd6c8958cdebaf5886494d52c7ac2df603f72adf6c7dc8ed9"}
Jul 02 21:22:42.772525 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.772213    2564 scope.go:117] "RemoveContainer" containerID="58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174"
Jul 02 21:22:42.780888 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.780869    2564 scope.go:117] "RemoveContainer" containerID="58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174"
Jul 02 21:22:42.781154 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:22:42.781135    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174\": container with ID starting with 58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174 not found: ID does not exist" containerID="58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174"
Jul 02 21:22:42.781205 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.781163    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174"} err="failed to get container status \"58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174\": rpc error: code = NotFound desc = could not find container \"58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174\": container with ID starting with 58887b58ffbec96a7fb02d96e14dacee41896d991f94c4834df64cbfdb6a7174 not found: ID does not exist"
Jul 02 21:22:42.793046 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.793022    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678dcd4695-tdbkn"]
Jul 02 21:22:42.797419 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:42.797400    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-678dcd4695-tdbkn"]
Jul 02 21:22:43.833248 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:22:43.833217    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c90d527-a551-43e4-a4e5-d3bf81532c46" path="/var/lib/kubelet/pods/1c90d527-a551-43e4-a4e5-d3bf81532c46/volumes"
Jul 02 21:23:29.580351 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:23:29.580298    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:23:51.610688 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:23:51.610659    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 21:24:50.041704 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:50.041610    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:51.178354 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.178316    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:24:51.575585 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.575551    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"]
Jul 02 21:24:51.575805 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.575792    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c90d527-a551-43e4-a4e5-d3bf81532c46" containerName="console"
Jul 02 21:24:51.575860 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.575807    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c90d527-a551-43e4-a4e5-d3bf81532c46" containerName="console"
Jul 02 21:24:51.575860 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.575858    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c90d527-a551-43e4-a4e5-d3bf81532c46" containerName="console"
Jul 02 21:24:51.578791 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.578769    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:24:51.582184 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.582163    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 02 21:24:51.583266 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.583247    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jx6cx\""
Jul 02 21:24:51.583359 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.583263    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 02 21:24:51.597648 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.597621    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"]
Jul 02 21:24:51.645353 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.645320    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:24:51.645503 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.645388    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:24:51.645503 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.645439    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l77d\" (UniqueName: \"kubernetes.io/projected/7caf8b54-936c-4f84-aa34-75ee29177fdc-kube-api-access-6l77d\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:24:51.746691 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.746653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:24:51.746691 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.746689    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l77d\" (UniqueName: \"kubernetes.io/projected/7caf8b54-936c-4f84-aa34-75ee29177fdc-kube-api-access-6l77d\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:24:51.746905 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.746813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:24:51.747078 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.747060    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:24:51.747169 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.747151    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:24:51.755425 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.755399    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l77d\" (UniqueName: \"kubernetes.io/projected/7caf8b54-936c-4f84-aa34-75ee29177fdc-kube-api-access-6l77d\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:24:51.886823 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:51.886731    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:24:52.014860 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:52.014838    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"]
Jul 02 21:24:52.017299 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:24:52.017271    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7caf8b54_936c_4f84_aa34_75ee29177fdc.slice/crio-3d944931a4ddffa2f817ff87aa8cde37e2b4a70d1b343dd5a30cf3e6f4e8c648 WatchSource:0}: Error finding container 3d944931a4ddffa2f817ff87aa8cde37e2b4a70d1b343dd5a30cf3e6f4e8c648: Status 404 returned error can't find the container with id 3d944931a4ddffa2f817ff87aa8cde37e2b4a70d1b343dd5a30cf3e6f4e8c648
Jul 02 21:24:52.019194 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:52.019177    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 21:24:52.188670 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:52.188584    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh" event={"ID":"7caf8b54-936c-4f84-aa34-75ee29177fdc","Type":"ContainerStarted","Data":"3d944931a4ddffa2f817ff87aa8cde37e2b4a70d1b343dd5a30cf3e6f4e8c648"}
Jul 02 21:24:59.217020 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:59.216975    2564 generic.go:356] "Generic (PLEG): container finished" podID="7caf8b54-936c-4f84-aa34-75ee29177fdc" containerID="e5fc566bf8401704b733ce2b9eeb07aaa5083b05ec69d06ce6951a3640233623" exitCode=0
Jul 02 21:24:59.217379 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:24:59.217057    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh" event={"ID":"7caf8b54-936c-4f84-aa34-75ee29177fdc","Type":"ContainerDied","Data":"e5fc566bf8401704b733ce2b9eeb07aaa5083b05ec69d06ce6951a3640233623"}
Jul 02 21:25:02.229654 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:02.229616    2564 generic.go:356] "Generic (PLEG): container finished" podID="7caf8b54-936c-4f84-aa34-75ee29177fdc" containerID="fd8145ab7b54db0af6853680585eb235719e28904756b16bae8449487cd4b0df" exitCode=0
Jul 02 21:25:02.230169 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:02.229676    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh" event={"ID":"7caf8b54-936c-4f84-aa34-75ee29177fdc","Type":"ContainerDied","Data":"fd8145ab7b54db0af6853680585eb235719e28904756b16bae8449487cd4b0df"}
Jul 02 21:25:02.953007 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:02.952952    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:03.031810 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:03.031774    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:09.258545 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:09.258510    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh" event={"ID":"7caf8b54-936c-4f84-aa34-75ee29177fdc","Type":"ContainerStarted","Data":"522d3d1963a2ce245451ab9273a048d90d0f93907521ba77df90b5682e870b3f"}
Jul 02 21:25:09.279518 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:09.279466    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh" podStartSLOduration=2.822539324 podStartE2EDuration="18.279448986s" totalImagesPullingTime="15.456909662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:24:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:25:09.278620978 +0000 UTC m=+378.012254310" watchObservedRunningTime="2026-07-02 21:25:09.279448986 +0000 UTC m=+378.013082293"
Jul 02 21:25:10.263751 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:10.263715    2564 generic.go:356] "Generic (PLEG): container finished" podID="7caf8b54-936c-4f84-aa34-75ee29177fdc" containerID="522d3d1963a2ce245451ab9273a048d90d0f93907521ba77df90b5682e870b3f" exitCode=0
Jul 02 21:25:10.264117 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:10.263795    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh" event={"ID":"7caf8b54-936c-4f84-aa34-75ee29177fdc","Type":"ContainerDied","Data":"522d3d1963a2ce245451ab9273a048d90d0f93907521ba77df90b5682e870b3f"}
Jul 02 21:25:11.387694 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:11.387670    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:25:11.510066 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:11.510024    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-bundle\") pod \"7caf8b54-936c-4f84-aa34-75ee29177fdc\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") "
Jul 02 21:25:11.510264 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:11.510121    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-util\") pod \"7caf8b54-936c-4f84-aa34-75ee29177fdc\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") "
Jul 02 21:25:11.510264 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:11.510184    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l77d\" (UniqueName: \"kubernetes.io/projected/7caf8b54-936c-4f84-aa34-75ee29177fdc-kube-api-access-6l77d\") pod \"7caf8b54-936c-4f84-aa34-75ee29177fdc\" (UID: \"7caf8b54-936c-4f84-aa34-75ee29177fdc\") "
Jul 02 21:25:11.510674 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:11.510650    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-bundle" (OuterVolumeSpecName: "bundle") pod "7caf8b54-936c-4f84-aa34-75ee29177fdc" (UID: "7caf8b54-936c-4f84-aa34-75ee29177fdc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:25:11.512378 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:11.512346    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7caf8b54-936c-4f84-aa34-75ee29177fdc-kube-api-access-6l77d" (OuterVolumeSpecName: "kube-api-access-6l77d") pod "7caf8b54-936c-4f84-aa34-75ee29177fdc" (UID: "7caf8b54-936c-4f84-aa34-75ee29177fdc"). InnerVolumeSpecName "kube-api-access-6l77d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:25:11.514385 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:11.514358    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-util" (OuterVolumeSpecName: "util") pod "7caf8b54-936c-4f84-aa34-75ee29177fdc" (UID: "7caf8b54-936c-4f84-aa34-75ee29177fdc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 21:25:11.610756 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:11.610670    2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-util\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:25:11.610756 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:11.610701    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l77d\" (UniqueName: \"kubernetes.io/projected/7caf8b54-936c-4f84-aa34-75ee29177fdc-kube-api-access-6l77d\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:25:11.610756 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:11.610711    2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7caf8b54-936c-4f84-aa34-75ee29177fdc-bundle\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:25:12.271751 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:12.271724    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh"
Jul 02 21:25:12.271751 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:12.271730    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gfdgh" event={"ID":"7caf8b54-936c-4f84-aa34-75ee29177fdc","Type":"ContainerDied","Data":"3d944931a4ddffa2f817ff87aa8cde37e2b4a70d1b343dd5a30cf3e6f4e8c648"}
Jul 02 21:25:12.271945 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:12.271797    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d944931a4ddffa2f817ff87aa8cde37e2b4a70d1b343dd5a30cf3e6f4e8c648"
Jul 02 21:25:13.325699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:13.325662    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:14.486511 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:14.486480    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:14.672450 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:14.672409    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:17.371175 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.371138    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"]
Jul 02 21:25:17.371552 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.371395    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7caf8b54-936c-4f84-aa34-75ee29177fdc" containerName="util"
Jul 02 21:25:17.371552 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.371406    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7caf8b54-936c-4f84-aa34-75ee29177fdc" containerName="util"
Jul 02 21:25:17.371552 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.371414    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7caf8b54-936c-4f84-aa34-75ee29177fdc" containerName="pull"
Jul 02 21:25:17.371552 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.371420    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7caf8b54-936c-4f84-aa34-75ee29177fdc" containerName="pull"
Jul 02 21:25:17.371552 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.371433    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7caf8b54-936c-4f84-aa34-75ee29177fdc" containerName="extract"
Jul 02 21:25:17.371552 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.371438    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7caf8b54-936c-4f84-aa34-75ee29177fdc" containerName="extract"
Jul 02 21:25:17.371552 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.371481    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7caf8b54-936c-4f84-aa34-75ee29177fdc" containerName="extract"
Jul 02 21:25:17.410246 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.410218    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"]
Jul 02 21:25:17.410383 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.410318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"
Jul 02 21:25:17.414721 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.414699    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 02 21:25:17.414951 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.414931    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 02 21:25:17.415085 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.415052    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-tq7sd\""
Jul 02 21:25:17.415195 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.415180    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 02 21:25:17.449313 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.449287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg4fs\" (UniqueName: \"kubernetes.io/projected/67fe9ac1-21a8-43d0-9dc3-04c25df79b4f-kube-api-access-lg4fs\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd\" (UID: \"67fe9ac1-21a8-43d0-9dc3-04c25df79b4f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"
Jul 02 21:25:17.449313 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.449315    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/67fe9ac1-21a8-43d0-9dc3-04c25df79b4f-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd\" (UID: \"67fe9ac1-21a8-43d0-9dc3-04c25df79b4f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"
Jul 02 21:25:17.550012 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.549969    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg4fs\" (UniqueName: \"kubernetes.io/projected/67fe9ac1-21a8-43d0-9dc3-04c25df79b4f-kube-api-access-lg4fs\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd\" (UID: \"67fe9ac1-21a8-43d0-9dc3-04c25df79b4f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"
Jul 02 21:25:17.550203 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.550034    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/67fe9ac1-21a8-43d0-9dc3-04c25df79b4f-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd\" (UID: \"67fe9ac1-21a8-43d0-9dc3-04c25df79b4f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"
Jul 02 21:25:17.552444 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.552421    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/67fe9ac1-21a8-43d0-9dc3-04c25df79b4f-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd\" (UID: \"67fe9ac1-21a8-43d0-9dc3-04c25df79b4f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"
Jul 02 21:25:17.564105 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.564083    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg4fs\" (UniqueName: \"kubernetes.io/projected/67fe9ac1-21a8-43d0-9dc3-04c25df79b4f-kube-api-access-lg4fs\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd\" (UID: \"67fe9ac1-21a8-43d0-9dc3-04c25df79b4f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"
Jul 02 21:25:17.720224 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.720192    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"
Jul 02 21:25:17.871163 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:17.871138    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"]
Jul 02 21:25:17.873267 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:25:17.873241    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fe9ac1_21a8_43d0_9dc3_04c25df79b4f.slice/crio-b9245d0e350bfac9869e27de43461f08c9eeac5dcbbb7ad645b6a3d4fc2b73cc WatchSource:0}: Error finding container b9245d0e350bfac9869e27de43461f08c9eeac5dcbbb7ad645b6a3d4fc2b73cc: Status 404 returned error can't find the container with id b9245d0e350bfac9869e27de43461f08c9eeac5dcbbb7ad645b6a3d4fc2b73cc
Jul 02 21:25:18.293329 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:18.293297    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd" event={"ID":"67fe9ac1-21a8-43d0-9dc3-04c25df79b4f","Type":"ContainerStarted","Data":"b9245d0e350bfac9869e27de43461f08c9eeac5dcbbb7ad645b6a3d4fc2b73cc"}
Jul 02 21:25:20.922541 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:20.922451    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:21.225444 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.225409    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-rkjhx"]
Jul 02 21:25:21.228544 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.228518    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:21.230453 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.230419    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:21.231193 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.231176    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 02 21:25:21.231409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.231395    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 02 21:25:21.231551 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.231532    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-8qm7v\""
Jul 02 21:25:21.243180 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.243158    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-rkjhx"]
Jul 02 21:25:21.285548 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.285517    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-cabundle0\") pod \"keda-operator-68bff59c-rkjhx\" (UID: \"7ad071ac-b29c-4dd6-9ad3-4168e607c97a\") " pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:21.285716 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.285558    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf7bb\" (UniqueName: \"kubernetes.io/projected/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-kube-api-access-qf7bb\") pod \"keda-operator-68bff59c-rkjhx\" (UID: \"7ad071ac-b29c-4dd6-9ad3-4168e607c97a\") " pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:21.285716 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.285592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-certificates\") pod \"keda-operator-68bff59c-rkjhx\" (UID: \"7ad071ac-b29c-4dd6-9ad3-4168e607c97a\") " pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:21.312703 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.312660    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd" event={"ID":"67fe9ac1-21a8-43d0-9dc3-04c25df79b4f","Type":"ContainerStarted","Data":"4c1482652539327d8fdfd1363fb69ee512a76510825510209b66dd4f0b247db4"}
Jul 02 21:25:21.312910 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.312849    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"
Jul 02 21:25:21.333713 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.333660    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd" podStartSLOduration=1.702258623 podStartE2EDuration="4.333640094s" totalImagesPullingTime="2.631381471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:25:21.333344606 +0000 UTC m=+390.066977934" watchObservedRunningTime="2026-07-02 21:25:21.333640094 +0000 UTC m=+390.067273404"
Jul 02 21:25:21.386784 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.386747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-cabundle0\") pod \"keda-operator-68bff59c-rkjhx\" (UID: \"7ad071ac-b29c-4dd6-9ad3-4168e607c97a\") " pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:21.386978 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.386798    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf7bb\" (UniqueName: \"kubernetes.io/projected/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-kube-api-access-qf7bb\") pod \"keda-operator-68bff59c-rkjhx\" (UID: \"7ad071ac-b29c-4dd6-9ad3-4168e607c97a\") " pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:21.386978 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.386850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-certificates\") pod \"keda-operator-68bff59c-rkjhx\" (UID: \"7ad071ac-b29c-4dd6-9ad3-4168e607c97a\") " pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:21.386978 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:25:21.386939    2564 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 02 21:25:21.386978 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:25:21.386956    2564 secret.go:281] references non-existent secret key: ca.crt
Jul 02 21:25:21.386978 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:25:21.386967    2564 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 02 21:25:21.386978 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:25:21.386984    2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-rkjhx: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 02 21:25:21.387235 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:25:21.387051    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-certificates podName:7ad071ac-b29c-4dd6-9ad3-4168e607c97a nodeName:}" failed. No retries permitted until 2026-07-02 21:25:21.88703467 +0000 UTC m=+390.620667969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-certificates") pod "keda-operator-68bff59c-rkjhx" (UID: "7ad071ac-b29c-4dd6-9ad3-4168e607c97a") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 02 21:25:21.387695 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.387674    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-cabundle0\") pod \"keda-operator-68bff59c-rkjhx\" (UID: \"7ad071ac-b29c-4dd6-9ad3-4168e607c97a\") " pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:21.400418 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.400386    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf7bb\" (UniqueName: \"kubernetes.io/projected/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-kube-api-access-qf7bb\") pod \"keda-operator-68bff59c-rkjhx\" (UID: \"7ad071ac-b29c-4dd6-9ad3-4168e607c97a\") " pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:21.481465 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.481360    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:21.598182 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.598135    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:21.675568 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.675537    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"]
Jul 02 21:25:21.678813 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.678795    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:21.684921 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.684897    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 02 21:25:21.698126 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.698091    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"]
Jul 02 21:25:21.790626 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.790539    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4e29e6af-327b-44c7-86a4-130068f7703f-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-msl5k\" (UID: \"4e29e6af-327b-44c7-86a4-130068f7703f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:21.790626 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.790592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpfm\" (UniqueName: \"kubernetes.io/projected/4e29e6af-327b-44c7-86a4-130068f7703f-kube-api-access-hnpfm\") pod \"keda-metrics-apiserver-559f75f947-msl5k\" (UID: \"4e29e6af-327b-44c7-86a4-130068f7703f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:21.790829 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.790686    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4e29e6af-327b-44c7-86a4-130068f7703f-certificates\") pod \"keda-metrics-apiserver-559f75f947-msl5k\" (UID: \"4e29e6af-327b-44c7-86a4-130068f7703f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:21.891234 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.891195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpfm\" (UniqueName: \"kubernetes.io/projected/4e29e6af-327b-44c7-86a4-130068f7703f-kube-api-access-hnpfm\") pod \"keda-metrics-apiserver-559f75f947-msl5k\" (UID: \"4e29e6af-327b-44c7-86a4-130068f7703f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:21.891396 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.891309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4e29e6af-327b-44c7-86a4-130068f7703f-certificates\") pod \"keda-metrics-apiserver-559f75f947-msl5k\" (UID: \"4e29e6af-327b-44c7-86a4-130068f7703f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:21.891396 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.891368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-certificates\") pod \"keda-operator-68bff59c-rkjhx\" (UID: \"7ad071ac-b29c-4dd6-9ad3-4168e607c97a\") " pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:21.891614 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.891408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4e29e6af-327b-44c7-86a4-130068f7703f-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-msl5k\" (UID: \"4e29e6af-327b-44c7-86a4-130068f7703f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:21.891851 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.891831    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4e29e6af-327b-44c7-86a4-130068f7703f-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-msl5k\" (UID: \"4e29e6af-327b-44c7-86a4-130068f7703f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:21.891910 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:25:21.891896    2564 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 02 21:25:21.891957 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:25:21.891916    2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-msl5k: secret "keda-metrics-apiserver-certs" not found
Jul 02 21:25:21.892028 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:25:21.891964    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e29e6af-327b-44c7-86a4-130068f7703f-certificates podName:4e29e6af-327b-44c7-86a4-130068f7703f nodeName:}" failed. No retries permitted until 2026-07-02 21:25:22.391947029 +0000 UTC m=+391.125580317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4e29e6af-327b-44c7-86a4-130068f7703f-certificates") pod "keda-metrics-apiserver-559f75f947-msl5k" (UID: "4e29e6af-327b-44c7-86a4-130068f7703f") : secret "keda-metrics-apiserver-certs" not found
Jul 02 21:25:21.893932 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.893908    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ad071ac-b29c-4dd6-9ad3-4168e607c97a-certificates\") pod \"keda-operator-68bff59c-rkjhx\" (UID: \"7ad071ac-b29c-4dd6-9ad3-4168e607c97a\") " pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:21.904442 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.904406    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpfm\" (UniqueName: \"kubernetes.io/projected/4e29e6af-327b-44c7-86a4-130068f7703f-kube-api-access-hnpfm\") pod \"keda-metrics-apiserver-559f75f947-msl5k\" (UID: \"4e29e6af-327b-44c7-86a4-130068f7703f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:21.950363 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.950313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:21.954210 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.954175    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-tp5zt"]
Jul 02 21:25:21.957985 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.957968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt"
Jul 02 21:25:21.960622 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.960597    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 02 21:25:21.971559 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.971536    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-tp5zt"]
Jul 02 21:25:21.992331 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.992301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fzg\" (UniqueName: \"kubernetes.io/projected/447eafb7-338c-4481-8e01-1d13ce7ab5a7-kube-api-access-m5fzg\") pod \"keda-admission-5fbd5c4644-tp5zt\" (UID: \"447eafb7-338c-4481-8e01-1d13ce7ab5a7\") " pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt"
Jul 02 21:25:21.992473 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:21.992451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/447eafb7-338c-4481-8e01-1d13ce7ab5a7-certificates\") pod \"keda-admission-5fbd5c4644-tp5zt\" (UID: \"447eafb7-338c-4481-8e01-1d13ce7ab5a7\") " pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt"
Jul 02 21:25:22.050763 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.050670    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:22.092950 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.092916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/447eafb7-338c-4481-8e01-1d13ce7ab5a7-certificates\") pod \"keda-admission-5fbd5c4644-tp5zt\" (UID: \"447eafb7-338c-4481-8e01-1d13ce7ab5a7\") " pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt"
Jul 02 21:25:22.093136 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.092966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fzg\" (UniqueName: \"kubernetes.io/projected/447eafb7-338c-4481-8e01-1d13ce7ab5a7-kube-api-access-m5fzg\") pod \"keda-admission-5fbd5c4644-tp5zt\" (UID: \"447eafb7-338c-4481-8e01-1d13ce7ab5a7\") " pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt"
Jul 02 21:25:22.093136 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:25:22.093094    2564 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 02 21:25:22.093136 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:25:22.093119    2564 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-tp5zt: secret "keda-admission-webhooks-certs" not found
Jul 02 21:25:22.093295 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:25:22.093170    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/447eafb7-338c-4481-8e01-1d13ce7ab5a7-certificates podName:447eafb7-338c-4481-8e01-1d13ce7ab5a7 nodeName:}" failed. No retries permitted until 2026-07-02 21:25:22.593155523 +0000 UTC m=+391.326788809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/447eafb7-338c-4481-8e01-1d13ce7ab5a7-certificates") pod "keda-admission-5fbd5c4644-tp5zt" (UID: "447eafb7-338c-4481-8e01-1d13ce7ab5a7") : secret "keda-admission-webhooks-certs" not found
Jul 02 21:25:22.103878 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.103852    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fzg\" (UniqueName: \"kubernetes.io/projected/447eafb7-338c-4481-8e01-1d13ce7ab5a7-kube-api-access-m5fzg\") pod \"keda-admission-5fbd5c4644-tp5zt\" (UID: \"447eafb7-338c-4481-8e01-1d13ce7ab5a7\") " pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt"
Jul 02 21:25:22.141225 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.141190    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:22.151256 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.151218    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:22.252109 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.252023    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:22.279969 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.279942    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-rkjhx"]
Jul 02 21:25:22.282501 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:25:22.282476    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad071ac_b29c_4dd6_9ad3_4168e607c97a.slice/crio-455e2e48e4c49da9a57438c584ce82775eb2645981ab0fd1d8fb088521b45f45 WatchSource:0}: Error finding container 455e2e48e4c49da9a57438c584ce82775eb2645981ab0fd1d8fb088521b45f45: Status 404 returned error can't find the container with id 455e2e48e4c49da9a57438c584ce82775eb2645981ab0fd1d8fb088521b45f45
Jul 02 21:25:22.318706 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.318598    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-rkjhx" event={"ID":"7ad071ac-b29c-4dd6-9ad3-4168e607c97a","Type":"ContainerStarted","Data":"455e2e48e4c49da9a57438c584ce82775eb2645981ab0fd1d8fb088521b45f45"}
Jul 02 21:25:22.385522 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.385490    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:22.396543 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.396514    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4e29e6af-327b-44c7-86a4-130068f7703f-certificates\") pod \"keda-metrics-apiserver-559f75f947-msl5k\" (UID: \"4e29e6af-327b-44c7-86a4-130068f7703f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:22.398882 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.398860    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4e29e6af-327b-44c7-86a4-130068f7703f-certificates\") pod \"keda-metrics-apiserver-559f75f947-msl5k\" (UID: \"4e29e6af-327b-44c7-86a4-130068f7703f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:22.537599 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.537563    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:22.589098 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.589014    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:22.598289 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.598263    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/447eafb7-338c-4481-8e01-1d13ce7ab5a7-certificates\") pod \"keda-admission-5fbd5c4644-tp5zt\" (UID: \"447eafb7-338c-4481-8e01-1d13ce7ab5a7\") " pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt"
Jul 02 21:25:22.600480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.600460    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/447eafb7-338c-4481-8e01-1d13ce7ab5a7-certificates\") pod \"keda-admission-5fbd5c4644-tp5zt\" (UID: \"447eafb7-338c-4481-8e01-1d13ce7ab5a7\") " pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt"
Jul 02 21:25:22.720093 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.720065    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"]
Jul 02 21:25:22.722437 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:25:22.722408    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e29e6af_327b_44c7_86a4_130068f7703f.slice/crio-cc985527e603bae0fd8d9599b5922b625d4a6a9a1e7c2c1fe0998c78a24d6ec3 WatchSource:0}: Error finding container cc985527e603bae0fd8d9599b5922b625d4a6a9a1e7c2c1fe0998c78a24d6ec3: Status 404 returned error can't find the container with id cc985527e603bae0fd8d9599b5922b625d4a6a9a1e7c2c1fe0998c78a24d6ec3
Jul 02 21:25:22.733940 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.733912    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:22.819210 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.819169    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:22.868166 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:22.868088    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt"
Jul 02 21:25:23.023361 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:23.023335    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-tp5zt"]
Jul 02 21:25:23.025157 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:25:23.025123    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447eafb7_338c_4481_8e01_1d13ce7ab5a7.slice/crio-4381409147fb99e22f695236efc955d0d0577c83393e5b6b5de184265d55ea02 WatchSource:0}: Error finding container 4381409147fb99e22f695236efc955d0d0577c83393e5b6b5de184265d55ea02: Status 404 returned error can't find the container with id 4381409147fb99e22f695236efc955d0d0577c83393e5b6b5de184265d55ea02
Jul 02 21:25:23.323605 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:23.323564    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt" event={"ID":"447eafb7-338c-4481-8e01-1d13ce7ab5a7","Type":"ContainerStarted","Data":"4381409147fb99e22f695236efc955d0d0577c83393e5b6b5de184265d55ea02"}
Jul 02 21:25:23.324694 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:23.324665    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k" event={"ID":"4e29e6af-327b-44c7-86a4-130068f7703f","Type":"ContainerStarted","Data":"cc985527e603bae0fd8d9599b5922b625d4a6a9a1e7c2c1fe0998c78a24d6ec3"}
Jul 02 21:25:25.339086 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:25.339047    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt" event={"ID":"447eafb7-338c-4481-8e01-1d13ce7ab5a7","Type":"ContainerStarted","Data":"9f8fbabd7c240351947efe68aa52dc5693574e3201acd44222c2af1bc43053ea"}
Jul 02 21:25:25.339540 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:25.339183    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt"
Jul 02 21:25:25.358531 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:25.358479    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt" podStartSLOduration=2.623811547 podStartE2EDuration="4.358465096s" totalImagesPullingTime="1.734653549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:25:25.357115616 +0000 UTC m=+394.090748946" watchObservedRunningTime="2026-07-02 21:25:25.358465096 +0000 UTC m=+394.092098404"
Jul 02 21:25:27.349478 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:27.349445    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k" event={"ID":"4e29e6af-327b-44c7-86a4-130068f7703f","Type":"ContainerStarted","Data":"897beaf43ae621277ae22a2db16b9f3273c23ccf9ead87d40da06aa43255e943"}
Jul 02 21:25:27.349926 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:27.349543    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:27.350855 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:27.350823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-rkjhx" event={"ID":"7ad071ac-b29c-4dd6-9ad3-4168e607c97a","Type":"ContainerStarted","Data":"173077b0f8999937a690642f8822a74fec5684abf6bc64a88ef6ec385ce8e35d"}
Jul 02 21:25:27.350988 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:27.350968    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:27.366831 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:27.366794    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k" podStartSLOduration=2.324016544 podStartE2EDuration="6.366781169s" totalImagesPullingTime="4.042764625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:25:27.365325698 +0000 UTC m=+396.098959013" watchObservedRunningTime="2026-07-02 21:25:27.366781169 +0000 UTC m=+396.100414454"
Jul 02 21:25:27.383326 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:27.383283    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-rkjhx" podStartSLOduration=1.8958679269999998 podStartE2EDuration="6.383270301s" totalImagesPullingTime="4.487402374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:25:27.38195448 +0000 UTC m=+396.115587788" watchObservedRunningTime="2026-07-02 21:25:27.383270301 +0000 UTC m=+396.116903609"
Jul 02 21:25:30.363701 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.363669    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-857f4fb655-8mdxx"]
Jul 02 21:25:30.367138 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.367117    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.379139 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.379112    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857f4fb655-8mdxx"]
Jul 02 21:25:30.406709 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.406681    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:30.466869 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.466836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-service-ca\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.466985 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.466877    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/905a5356-350d-49b4-8147-40238b85b1d9-console-oauth-config\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.466985 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.466959    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-console-config\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.467099 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.467018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/905a5356-350d-49b4-8147-40238b85b1d9-console-serving-cert\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.467099 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.467066    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-trusted-ca-bundle\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.467099 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.467095    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-oauth-serving-cert\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.467191 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.467130    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h224x\" (UniqueName: \"kubernetes.io/projected/905a5356-350d-49b4-8147-40238b85b1d9-kube-api-access-h224x\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.568284 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.568248    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-service-ca\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.568284 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.568289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/905a5356-350d-49b4-8147-40238b85b1d9-console-oauth-config\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.568479 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.568319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-console-config\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.568479 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.568385    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/905a5356-350d-49b4-8147-40238b85b1d9-console-serving-cert\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.568479 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.568428    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-trusted-ca-bundle\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.568479 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.568451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-oauth-serving-cert\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.568479 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.568471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h224x\" (UniqueName: \"kubernetes.io/projected/905a5356-350d-49b4-8147-40238b85b1d9-kube-api-access-h224x\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.569088 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.569061    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-service-ca\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.569220 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.569129    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-console-config\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.569270 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.569230    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-trusted-ca-bundle\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.569302 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.569259    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/905a5356-350d-49b4-8147-40238b85b1d9-oauth-serving-cert\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.570815 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.570792    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/905a5356-350d-49b4-8147-40238b85b1d9-console-oauth-config\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.570975 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.570958    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/905a5356-350d-49b4-8147-40238b85b1d9-console-serving-cert\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.576357 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.576339    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h224x\" (UniqueName: \"kubernetes.io/projected/905a5356-350d-49b4-8147-40238b85b1d9-kube-api-access-h224x\") pod \"console-857f4fb655-8mdxx\" (UID: \"905a5356-350d-49b4-8147-40238b85b1d9\") " pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.676108 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.676019    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:30.803957 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:30.803917    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857f4fb655-8mdxx"]
Jul 02 21:25:30.806724 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:25:30.806690    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905a5356_350d_49b4_8147_40238b85b1d9.slice/crio-20d495e7c91e58d5248485a10b52644b705719cadd2bf4921bbb7549c9d65480 WatchSource:0}: Error finding container 20d495e7c91e58d5248485a10b52644b705719cadd2bf4921bbb7549c9d65480: Status 404 returned error can't find the container with id 20d495e7c91e58d5248485a10b52644b705719cadd2bf4921bbb7549c9d65480
Jul 02 21:25:31.368091 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:31.368044    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857f4fb655-8mdxx" event={"ID":"905a5356-350d-49b4-8147-40238b85b1d9","Type":"ContainerStarted","Data":"4a84034f9620c9cc48eede77f0adc2293c7b624e853ea715cd8eb3d639750dbc"}
Jul 02 21:25:31.368518 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:31.368107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857f4fb655-8mdxx" event={"ID":"905a5356-350d-49b4-8147-40238b85b1d9","Type":"ContainerStarted","Data":"20d495e7c91e58d5248485a10b52644b705719cadd2bf4921bbb7549c9d65480"}
Jul 02 21:25:31.387412 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:31.387364    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-857f4fb655-8mdxx" podStartSLOduration=1.387348676 podStartE2EDuration="1.387348676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:25:31.38565651 +0000 UTC m=+400.119289829" watchObservedRunningTime="2026-07-02 21:25:31.387348676 +0000 UTC m=+400.120981984"
Jul 02 21:25:38.358577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:38.358547    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-msl5k"
Jul 02 21:25:40.676969 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:40.676928    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:40.677380 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:40.677003    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:40.681938 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:40.681914    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:41.412086 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:41.412057    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-857f4fb655-8mdxx"
Jul 02 21:25:41.458125 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:41.458087    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b674bdc77-xn64j"]
Jul 02 21:25:41.623589 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:41.623549    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:41.723671 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:41.723637    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:25:42.321474 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:42.321444    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-4zzcd"
Jul 02 21:25:46.346752 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:46.346719    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-tp5zt"
Jul 02 21:25:48.357117 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:48.357085    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-rkjhx"
Jul 02 21:25:58.905417 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:25:58.905379    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:06.478226 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.478182    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-b674bdc77-xn64j" podUID="f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" containerName="console" containerID="cri-o://d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452" gracePeriod=15
Jul 02 21:26:06.728782 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.728730    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:26:06.866342 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.866309    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-config\") pod \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") "
Jul 02 21:26:06.866342 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.866344    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-775r6\" (UniqueName: \"kubernetes.io/projected/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-kube-api-access-775r6\") pod \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") "
Jul 02 21:26:06.866544 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.866479    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-oauth-config\") pod \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") "
Jul 02 21:26:06.866544 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.866513    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-oauth-serving-cert\") pod \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") "
Jul 02 21:26:06.866544 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.866532    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-service-ca\") pod \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") "
Jul 02 21:26:06.866650 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.866566    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-serving-cert\") pod \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") "
Jul 02 21:26:06.866650 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.866605    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-trusted-ca-bundle\") pod \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\" (UID: \"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2\") "
Jul 02 21:26:06.866863 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.866827    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-config" (OuterVolumeSpecName: "console-config") pod "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" (UID: "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:26:06.866976 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.866880    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" (UID: "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:26:06.866976 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.866895    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-service-ca" (OuterVolumeSpecName: "service-ca") pod "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" (UID: "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:26:06.867150 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.867127    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" (UID: "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:26:06.868556 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.868531    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" (UID: "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:26:06.868658 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.868563    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" (UID: "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:26:06.868658 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.868618    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-kube-api-access-775r6" (OuterVolumeSpecName: "kube-api-access-775r6") pod "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" (UID: "f4d7fdf5-3fee-40fb-ac9a-742932cf55a2"). InnerVolumeSpecName "kube-api-access-775r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:26:06.967794 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.967758    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-oauth-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:26:06.967794 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.967787    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-oauth-serving-cert\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:26:06.967794 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.967801    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-service-ca\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:26:06.968039 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.967813    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-serving-cert\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:26:06.968039 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.967824    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-trusted-ca-bundle\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:26:06.968039 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.967835    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-console-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:26:06.968039 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:06.967846    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-775r6\" (UniqueName: \"kubernetes.io/projected/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2-kube-api-access-775r6\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:26:07.515286 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.515250    2564 generic.go:356] "Generic (PLEG): container finished" podID="f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" containerID="d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452" exitCode=0
Jul 02 21:26:07.515715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.515310    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b674bdc77-xn64j"
Jul 02 21:26:07.515715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.515344    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b674bdc77-xn64j" event={"ID":"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2","Type":"ContainerDied","Data":"d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452"}
Jul 02 21:26:07.515715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.515413    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b674bdc77-xn64j" event={"ID":"f4d7fdf5-3fee-40fb-ac9a-742932cf55a2","Type":"ContainerDied","Data":"1585fa177daf53d83b66beecde734a271a9787f5a9bf8d2f67ef46b1b45a15b3"}
Jul 02 21:26:07.515715 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.515432    2564 scope.go:117] "RemoveContainer" containerID="d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452"
Jul 02 21:26:07.523562 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.523546    2564 scope.go:117] "RemoveContainer" containerID="d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452"
Jul 02 21:26:07.523813 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:26:07.523794    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452\": container with ID starting with d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452 not found: ID does not exist" containerID="d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452"
Jul 02 21:26:07.523871 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.523826    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452"} err="failed to get container status \"d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452\": rpc error: code = NotFound desc = could not find container \"d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452\": container with ID starting with d0bb9a92ec63a5221ca0826dd5eb8d182ac24b86dbab8a8dee248b04e4d50452 not found: ID does not exist"
Jul 02 21:26:07.536887 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.536865    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b674bdc77-xn64j"]
Jul 02 21:26:07.543579 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.543555    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b674bdc77-xn64j"]
Jul 02 21:26:07.674480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.674439    2564 patch_prober.go:28] interesting pod/console-b674bdc77-xn64j container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 02 21:26:07.674659 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.674506    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-b674bdc77-xn64j" podUID="f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" containerName="console" probeResult="failure" output="Get \"https://10.133.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 02 21:26:07.834250 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:07.834169    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" path="/var/lib/kubelet/pods/f4d7fdf5-3fee-40fb-ac9a-742932cf55a2/volumes"
Jul 02 21:26:26.915952 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.915914    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:26.980131 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.980099    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-wndjb"]
Jul 02 21:26:26.980504 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.980485    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" containerName="console"
Jul 02 21:26:26.980504 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.980504    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" containerName="console"
Jul 02 21:26:26.980638 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.980554    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4d7fdf5-3fee-40fb-ac9a-742932cf55a2" containerName="console"
Jul 02 21:26:26.983515 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.983491    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:26:26.985659 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.985631    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:26.986050 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.986019    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 21:26:26.986177 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.986060    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 02 21:26:26.986245 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.986187    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 21:26:26.987251 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.987230    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-mnqcb\""
Jul 02 21:26:26.992338 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:26.992312    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-wndjb"]
Jul 02 21:26:27.015257 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.015227    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bfpf\" (UniqueName: \"kubernetes.io/projected/78e255c1-4024-463b-8da1-389142535c21-kube-api-access-4bfpf\") pod \"kserve-controller-manager-56668bddcf-wndjb\" (UID: \"78e255c1-4024-463b-8da1-389142535c21\") " pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:26:27.015458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.015277    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78e255c1-4024-463b-8da1-389142535c21-cert\") pod \"kserve-controller-manager-56668bddcf-wndjb\" (UID: \"78e255c1-4024-463b-8da1-389142535c21\") " pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:26:27.020832 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.020787    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-tfm28"]
Jul 02 21:26:27.023973 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.023950    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tfm28"
Jul 02 21:26:27.027120 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.027095    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mc9tx\""
Jul 02 21:26:27.027291 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.027272    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 21:26:27.041395 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.041370    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tfm28"]
Jul 02 21:26:27.115781 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.115742    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78e255c1-4024-463b-8da1-389142535c21-cert\") pod \"kserve-controller-manager-56668bddcf-wndjb\" (UID: \"78e255c1-4024-463b-8da1-389142535c21\") " pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:26:27.115956 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.115801    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggs2l\" (UniqueName: \"kubernetes.io/projected/8758349f-2525-492a-b49c-af4e553ec097-kube-api-access-ggs2l\") pod \"seaweedfs-64568bcd49-tfm28\" (UID: \"8758349f-2525-492a-b49c-af4e553ec097\") " pod="kserve/seaweedfs-64568bcd49-tfm28"
Jul 02 21:26:27.115956 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.115833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8758349f-2525-492a-b49c-af4e553ec097-data\") pod \"seaweedfs-64568bcd49-tfm28\" (UID: \"8758349f-2525-492a-b49c-af4e553ec097\") " pod="kserve/seaweedfs-64568bcd49-tfm28"
Jul 02 21:26:27.115956 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.115887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bfpf\" (UniqueName: \"kubernetes.io/projected/78e255c1-4024-463b-8da1-389142535c21-kube-api-access-4bfpf\") pod \"kserve-controller-manager-56668bddcf-wndjb\" (UID: \"78e255c1-4024-463b-8da1-389142535c21\") " pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:26:27.115956 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:26:27.115944    2564 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 02 21:26:27.116163 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:26:27.116017    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e255c1-4024-463b-8da1-389142535c21-cert podName:78e255c1-4024-463b-8da1-389142535c21 nodeName:}" failed. No retries permitted until 2026-07-02 21:26:27.616000972 +0000 UTC m=+456.349634271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/78e255c1-4024-463b-8da1-389142535c21-cert") pod "kserve-controller-manager-56668bddcf-wndjb" (UID: "78e255c1-4024-463b-8da1-389142535c21") : secret "kserve-webhook-server-cert" not found
Jul 02 21:26:27.126741 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.126708    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bfpf\" (UniqueName: \"kubernetes.io/projected/78e255c1-4024-463b-8da1-389142535c21-kube-api-access-4bfpf\") pod \"kserve-controller-manager-56668bddcf-wndjb\" (UID: \"78e255c1-4024-463b-8da1-389142535c21\") " pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:26:27.147339 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.147313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:27.216322 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.216296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggs2l\" (UniqueName: \"kubernetes.io/projected/8758349f-2525-492a-b49c-af4e553ec097-kube-api-access-ggs2l\") pod \"seaweedfs-64568bcd49-tfm28\" (UID: \"8758349f-2525-492a-b49c-af4e553ec097\") " pod="kserve/seaweedfs-64568bcd49-tfm28"
Jul 02 21:26:27.216462 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.216331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8758349f-2525-492a-b49c-af4e553ec097-data\") pod \"seaweedfs-64568bcd49-tfm28\" (UID: \"8758349f-2525-492a-b49c-af4e553ec097\") " pod="kserve/seaweedfs-64568bcd49-tfm28"
Jul 02 21:26:27.216675 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.216659    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8758349f-2525-492a-b49c-af4e553ec097-data\") pod \"seaweedfs-64568bcd49-tfm28\" (UID: \"8758349f-2525-492a-b49c-af4e553ec097\") " pod="kserve/seaweedfs-64568bcd49-tfm28"
Jul 02 21:26:27.225950 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.225919    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggs2l\" (UniqueName: \"kubernetes.io/projected/8758349f-2525-492a-b49c-af4e553ec097-kube-api-access-ggs2l\") pod \"seaweedfs-64568bcd49-tfm28\" (UID: \"8758349f-2525-492a-b49c-af4e553ec097\") " pod="kserve/seaweedfs-64568bcd49-tfm28"
Jul 02 21:26:27.249245 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.249211    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:27.333294 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.333265    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tfm28"
Jul 02 21:26:27.347750 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.347693    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:27.463871 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.463846    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tfm28"]
Jul 02 21:26:27.466413 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:26:27.466349    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8758349f_2525_492a_b49c_af4e553ec097.slice/crio-4368de10cdcb9ec88a68bc051cfda97ac9200edbf49c5fa74a1a549b5fe3b14e WatchSource:0}: Error finding container 4368de10cdcb9ec88a68bc051cfda97ac9200edbf49c5fa74a1a549b5fe3b14e: Status 404 returned error can't find the container with id 4368de10cdcb9ec88a68bc051cfda97ac9200edbf49c5fa74a1a549b5fe3b14e
Jul 02 21:26:27.593301 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.593268    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tfm28" event={"ID":"8758349f-2525-492a-b49c-af4e553ec097","Type":"ContainerStarted","Data":"4368de10cdcb9ec88a68bc051cfda97ac9200edbf49c5fa74a1a549b5fe3b14e"}
Jul 02 21:26:27.619152 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.619125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78e255c1-4024-463b-8da1-389142535c21-cert\") pod \"kserve-controller-manager-56668bddcf-wndjb\" (UID: \"78e255c1-4024-463b-8da1-389142535c21\") " pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:26:27.621415 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.621394    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78e255c1-4024-463b-8da1-389142535c21-cert\") pod \"kserve-controller-manager-56668bddcf-wndjb\" (UID: \"78e255c1-4024-463b-8da1-389142535c21\") " pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:26:27.816633 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.816549    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:27.895928 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.895895    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:26:27.917823 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:27.917781    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:26:28.035127 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:28.035046    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-wndjb"]
Jul 02 21:26:28.039725 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:26:28.039674    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e255c1_4024_463b_8da1_389142535c21.slice/crio-1a252bed888839c2f18188bfcd0059ea5b7346ad7bf38e5af9b099507fbb0e13 WatchSource:0}: Error finding container 1a252bed888839c2f18188bfcd0059ea5b7346ad7bf38e5af9b099507fbb0e13: Status 404 returned error can't find the container with id 1a252bed888839c2f18188bfcd0059ea5b7346ad7bf38e5af9b099507fbb0e13
Jul 02 21:26:28.599458 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:28.599415    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-wndjb" event={"ID":"78e255c1-4024-463b-8da1-389142535c21","Type":"ContainerStarted","Data":"1a252bed888839c2f18188bfcd0059ea5b7346ad7bf38e5af9b099507fbb0e13"}
Jul 02 21:26:32.617513 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:32.617480    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tfm28" event={"ID":"8758349f-2525-492a-b49c-af4e553ec097","Type":"ContainerStarted","Data":"fb2ed282e5ade95094d6b7ed6402029e8034f35322a1512ab80b1e8e58f9ec85"}
Jul 02 21:26:32.617977 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:32.617532    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-tfm28"
Jul 02 21:26:32.618878 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:32.618854    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-wndjb" event={"ID":"78e255c1-4024-463b-8da1-389142535c21","Type":"ContainerStarted","Data":"f1118703ef3fe2e7794463d4a4e8fbea69661d94dd7aef9c857f76e396db835d"}
Jul 02 21:26:32.619018 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:32.618982    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:26:32.634465 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:32.634418    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-tfm28" podStartSLOduration=2.351740416 podStartE2EDuration="6.634404829s" totalImagesPullingTime="4.282664413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:26:32.633738501 +0000 UTC m=+461.367371833" watchObservedRunningTime="2026-07-02 21:26:32.634404829 +0000 UTC m=+461.368038115"
Jul 02 21:26:32.650099 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:32.650038    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-56668bddcf-wndjb" podStartSLOduration=2.997900337 podStartE2EDuration="6.650022877s" totalImagesPullingTime="3.65212254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:26:32.649757748 +0000 UTC m=+461.383391055" watchObservedRunningTime="2026-07-02 21:26:32.650022877 +0000 UTC m=+461.383656182"
Jul 02 21:26:38.625087 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:26:38.625053    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-tfm28"
Jul 02 21:27:03.564449 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.564413    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-wndjb"]
Jul 02 21:27:03.564914 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.564625    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-56668bddcf-wndjb" podUID="78e255c1-4024-463b-8da1-389142535c21" containerName="manager" containerID="cri-o://f1118703ef3fe2e7794463d4a4e8fbea69661d94dd7aef9c857f76e396db835d" gracePeriod=10
Jul 02 21:27:03.569426 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.569394    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:27:03.596687 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.596662    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-fpqmb"]
Jul 02 21:27:03.599818 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.599800    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-fpqmb"
Jul 02 21:27:03.609854 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.609831    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-fpqmb"]
Jul 02 21:27:03.682763 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.682730    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3816d407-592b-43fe-9219-e516cdfc949c-cert\") pod \"kserve-controller-manager-56668bddcf-fpqmb\" (UID: \"3816d407-592b-43fe-9219-e516cdfc949c\") " pod="kserve/kserve-controller-manager-56668bddcf-fpqmb"
Jul 02 21:27:03.682889 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.682782    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbdgc\" (UniqueName: \"kubernetes.io/projected/3816d407-592b-43fe-9219-e516cdfc949c-kube-api-access-xbdgc\") pod \"kserve-controller-manager-56668bddcf-fpqmb\" (UID: \"3816d407-592b-43fe-9219-e516cdfc949c\") " pod="kserve/kserve-controller-manager-56668bddcf-fpqmb"
Jul 02 21:27:03.738963 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.738927    2564 generic.go:356] "Generic (PLEG): container finished" podID="78e255c1-4024-463b-8da1-389142535c21" containerID="f1118703ef3fe2e7794463d4a4e8fbea69661d94dd7aef9c857f76e396db835d" exitCode=0
Jul 02 21:27:03.739167 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.739036    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-wndjb" event={"ID":"78e255c1-4024-463b-8da1-389142535c21","Type":"ContainerDied","Data":"f1118703ef3fe2e7794463d4a4e8fbea69661d94dd7aef9c857f76e396db835d"}
Jul 02 21:27:03.783865 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.783830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3816d407-592b-43fe-9219-e516cdfc949c-cert\") pod \"kserve-controller-manager-56668bddcf-fpqmb\" (UID: \"3816d407-592b-43fe-9219-e516cdfc949c\") " pod="kserve/kserve-controller-manager-56668bddcf-fpqmb"
Jul 02 21:27:03.783972 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.783875    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbdgc\" (UniqueName: \"kubernetes.io/projected/3816d407-592b-43fe-9219-e516cdfc949c-kube-api-access-xbdgc\") pod \"kserve-controller-manager-56668bddcf-fpqmb\" (UID: \"3816d407-592b-43fe-9219-e516cdfc949c\") " pod="kserve/kserve-controller-manager-56668bddcf-fpqmb"
Jul 02 21:27:03.786063 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.786042    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3816d407-592b-43fe-9219-e516cdfc949c-cert\") pod \"kserve-controller-manager-56668bddcf-fpqmb\" (UID: \"3816d407-592b-43fe-9219-e516cdfc949c\") " pod="kserve/kserve-controller-manager-56668bddcf-fpqmb"
Jul 02 21:27:03.793377 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.793359    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbdgc\" (UniqueName: \"kubernetes.io/projected/3816d407-592b-43fe-9219-e516cdfc949c-kube-api-access-xbdgc\") pod \"kserve-controller-manager-56668bddcf-fpqmb\" (UID: \"3816d407-592b-43fe-9219-e516cdfc949c\") " pod="kserve/kserve-controller-manager-56668bddcf-fpqmb"
Jul 02 21:27:03.824624 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.824576    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:27:03.884410 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.884381    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bfpf\" (UniqueName: \"kubernetes.io/projected/78e255c1-4024-463b-8da1-389142535c21-kube-api-access-4bfpf\") pod \"78e255c1-4024-463b-8da1-389142535c21\" (UID: \"78e255c1-4024-463b-8da1-389142535c21\") "
Jul 02 21:27:03.884570 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.884431    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78e255c1-4024-463b-8da1-389142535c21-cert\") pod \"78e255c1-4024-463b-8da1-389142535c21\" (UID: \"78e255c1-4024-463b-8da1-389142535c21\") "
Jul 02 21:27:03.886471 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.886446    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e255c1-4024-463b-8da1-389142535c21-kube-api-access-4bfpf" (OuterVolumeSpecName: "kube-api-access-4bfpf") pod "78e255c1-4024-463b-8da1-389142535c21" (UID: "78e255c1-4024-463b-8da1-389142535c21"). InnerVolumeSpecName "kube-api-access-4bfpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:27:03.886746 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.886725    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e255c1-4024-463b-8da1-389142535c21-cert" (OuterVolumeSpecName: "cert") pod "78e255c1-4024-463b-8da1-389142535c21" (UID: "78e255c1-4024-463b-8da1-389142535c21"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:27:03.953837 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.953811    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-fpqmb"
Jul 02 21:27:03.985218 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.985201    2564 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78e255c1-4024-463b-8da1-389142535c21-cert\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:27:03.985218 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:03.985220    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bfpf\" (UniqueName: \"kubernetes.io/projected/78e255c1-4024-463b-8da1-389142535c21-kube-api-access-4bfpf\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:27:04.102718 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:04.102684    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-fpqmb"]
Jul 02 21:27:04.104790 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:27:04.104763    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3816d407_592b_43fe_9219_e516cdfc949c.slice/crio-0e0826583d0716ec49aca97fa0935b420a051f0aa0165c38a8bbf27178d1552f WatchSource:0}: Error finding container 0e0826583d0716ec49aca97fa0935b420a051f0aa0165c38a8bbf27178d1552f: Status 404 returned error can't find the container with id 0e0826583d0716ec49aca97fa0935b420a051f0aa0165c38a8bbf27178d1552f
Jul 02 21:27:04.744978 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:04.744952    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-56668bddcf-wndjb"
Jul 02 21:27:04.745388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:04.744971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-wndjb" event={"ID":"78e255c1-4024-463b-8da1-389142535c21","Type":"ContainerDied","Data":"1a252bed888839c2f18188bfcd0059ea5b7346ad7bf38e5af9b099507fbb0e13"}
Jul 02 21:27:04.745388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:04.745048    2564 scope.go:117] "RemoveContainer" containerID="f1118703ef3fe2e7794463d4a4e8fbea69661d94dd7aef9c857f76e396db835d"
Jul 02 21:27:04.746768 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:04.746738    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-fpqmb" event={"ID":"3816d407-592b-43fe-9219-e516cdfc949c","Type":"ContainerStarted","Data":"eb6d6a2d3dfc1fd5dce7e1775b3742cae46b0e148b733b14c2bf1ab60225c5d9"}
Jul 02 21:27:04.746877 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:04.746782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-56668bddcf-fpqmb" event={"ID":"3816d407-592b-43fe-9219-e516cdfc949c","Type":"ContainerStarted","Data":"0e0826583d0716ec49aca97fa0935b420a051f0aa0165c38a8bbf27178d1552f"}
Jul 02 21:27:04.746877 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:04.746860    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-56668bddcf-fpqmb"
Jul 02 21:27:04.766105 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:04.766063    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-56668bddcf-fpqmb" podStartSLOduration=1.248198077 podStartE2EDuration="1.766049058s" totalImagesPullingTime="517.850981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:27:04.765115801 +0000 UTC m=+493.498749107" watchObservedRunningTime="2026-07-02 21:27:04.766049058 +0000 UTC m=+493.499682363"
Jul 02 21:27:04.779460 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:04.779439    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-wndjb"]
Jul 02 21:27:04.784171 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:04.784150    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-56668bddcf-wndjb"]
Jul 02 21:27:05.832970 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:05.832938    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78e255c1-4024-463b-8da1-389142535c21" path="/var/lib/kubelet/pods/78e255c1-4024-463b-8da1-389142535c21/volumes"
Jul 02 21:27:09.067903 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:09.067877    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:35.756870 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:35.756839    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-56668bddcf-fpqmb"
Jul 02 21:27:36.923806 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.923772    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:36.966622 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.966567    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-h7bw7"]
Jul 02 21:27:36.966928 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.966911    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e255c1-4024-463b-8da1-389142535c21" containerName="manager"
Jul 02 21:27:36.966979 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.966930    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e255c1-4024-463b-8da1-389142535c21" containerName="manager"
Jul 02 21:27:36.967038 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.967006    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e255c1-4024-463b-8da1-389142535c21" containerName="manager"
Jul 02 21:27:36.969504 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.969490    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-h7bw7"
Jul 02 21:27:36.973285 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.973258    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-xdsmq\""
Jul 02 21:27:36.973388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.973283    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 02 21:27:36.981113 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.981093    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-jtqwk"]
Jul 02 21:27:36.984564 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.984546    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-jtqwk"
Jul 02 21:27:36.985249 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.985230    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-h7bw7"]
Jul 02 21:27:36.987860 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.987831    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-wc68b\""
Jul 02 21:27:36.988257 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.988240    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 02 21:27:36.994518 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:36.994491    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-jtqwk"]
Jul 02 21:27:37.057964 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.057934    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:37.147932 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.147896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dca3ea7-71e8-4657-a64d-fe18841eac6f-tls-certs\") pod \"model-serving-api-fd65d7d6-h7bw7\" (UID: \"6dca3ea7-71e8-4657-a64d-fe18841eac6f\") " pod="kserve/model-serving-api-fd65d7d6-h7bw7"
Jul 02 21:27:37.148103 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.147941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf9gq\" (UniqueName: \"kubernetes.io/projected/e833578a-9fa9-440d-a13d-197c1043b539-kube-api-access-mf9gq\") pod \"odh-model-controller-84b54b7f97-jtqwk\" (UID: \"e833578a-9fa9-440d-a13d-197c1043b539\") " pod="kserve/odh-model-controller-84b54b7f97-jtqwk"
Jul 02 21:27:37.148103 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.148034    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhfc2\" (UniqueName: \"kubernetes.io/projected/6dca3ea7-71e8-4657-a64d-fe18841eac6f-kube-api-access-lhfc2\") pod \"model-serving-api-fd65d7d6-h7bw7\" (UID: \"6dca3ea7-71e8-4657-a64d-fe18841eac6f\") " pod="kserve/model-serving-api-fd65d7d6-h7bw7"
Jul 02 21:27:37.148103 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.148064    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e833578a-9fa9-440d-a13d-197c1043b539-cert\") pod \"odh-model-controller-84b54b7f97-jtqwk\" (UID: \"e833578a-9fa9-440d-a13d-197c1043b539\") " pod="kserve/odh-model-controller-84b54b7f97-jtqwk"
Jul 02 21:27:37.158585 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.158557    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:37.248743 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.248714    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhfc2\" (UniqueName: \"kubernetes.io/projected/6dca3ea7-71e8-4657-a64d-fe18841eac6f-kube-api-access-lhfc2\") pod \"model-serving-api-fd65d7d6-h7bw7\" (UID: \"6dca3ea7-71e8-4657-a64d-fe18841eac6f\") " pod="kserve/model-serving-api-fd65d7d6-h7bw7"
Jul 02 21:27:37.248890 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.248754    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e833578a-9fa9-440d-a13d-197c1043b539-cert\") pod \"odh-model-controller-84b54b7f97-jtqwk\" (UID: \"e833578a-9fa9-440d-a13d-197c1043b539\") " pod="kserve/odh-model-controller-84b54b7f97-jtqwk"
Jul 02 21:27:37.248890 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.248796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dca3ea7-71e8-4657-a64d-fe18841eac6f-tls-certs\") pod \"model-serving-api-fd65d7d6-h7bw7\" (UID: \"6dca3ea7-71e8-4657-a64d-fe18841eac6f\") " pod="kserve/model-serving-api-fd65d7d6-h7bw7"
Jul 02 21:27:37.248890 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.248841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf9gq\" (UniqueName: \"kubernetes.io/projected/e833578a-9fa9-440d-a13d-197c1043b539-kube-api-access-mf9gq\") pod \"odh-model-controller-84b54b7f97-jtqwk\" (UID: \"e833578a-9fa9-440d-a13d-197c1043b539\") " pod="kserve/odh-model-controller-84b54b7f97-jtqwk"
Jul 02 21:27:37.251290 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.251267    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dca3ea7-71e8-4657-a64d-fe18841eac6f-tls-certs\") pod \"model-serving-api-fd65d7d6-h7bw7\" (UID: \"6dca3ea7-71e8-4657-a64d-fe18841eac6f\") " pod="kserve/model-serving-api-fd65d7d6-h7bw7"
Jul 02 21:27:37.251401 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.251353    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e833578a-9fa9-440d-a13d-197c1043b539-cert\") pod \"odh-model-controller-84b54b7f97-jtqwk\" (UID: \"e833578a-9fa9-440d-a13d-197c1043b539\") " pod="kserve/odh-model-controller-84b54b7f97-jtqwk"
Jul 02 21:27:37.257839 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.257813    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhfc2\" (UniqueName: \"kubernetes.io/projected/6dca3ea7-71e8-4657-a64d-fe18841eac6f-kube-api-access-lhfc2\") pod \"model-serving-api-fd65d7d6-h7bw7\" (UID: \"6dca3ea7-71e8-4657-a64d-fe18841eac6f\") " pod="kserve/model-serving-api-fd65d7d6-h7bw7"
Jul 02 21:27:37.257934 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.257818    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf9gq\" (UniqueName: \"kubernetes.io/projected/e833578a-9fa9-440d-a13d-197c1043b539-kube-api-access-mf9gq\") pod \"odh-model-controller-84b54b7f97-jtqwk\" (UID: \"e833578a-9fa9-440d-a13d-197c1043b539\") " pod="kserve/odh-model-controller-84b54b7f97-jtqwk"
Jul 02 21:27:37.280572 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.280553    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-h7bw7"
Jul 02 21:27:37.301159 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.301133    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-jtqwk"
Jul 02 21:27:37.417512 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.417480    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-h7bw7"]
Jul 02 21:27:37.419312 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:27:37.419281    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dca3ea7_71e8_4657_a64d_fe18841eac6f.slice/crio-9628ff88e681d2a7fe4ba402dce624ab0b4ffd31fe463f4fb9cd00ea61af520d WatchSource:0}: Error finding container 9628ff88e681d2a7fe4ba402dce624ab0b4ffd31fe463f4fb9cd00ea61af520d: Status 404 returned error can't find the container with id 9628ff88e681d2a7fe4ba402dce624ab0b4ffd31fe463f4fb9cd00ea61af520d
Jul 02 21:27:37.442285 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.442261    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-jtqwk"]
Jul 02 21:27:37.445893 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:27:37.445863    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode833578a_9fa9_440d_a13d_197c1043b539.slice/crio-7a8448c2363fc10c896e4723f4480586f1b8addcfd800a94ce03558f808461a6 WatchSource:0}: Error finding container 7a8448c2363fc10c896e4723f4480586f1b8addcfd800a94ce03558f808461a6: Status 404 returned error can't find the container with id 7a8448c2363fc10c896e4723f4480586f1b8addcfd800a94ce03558f808461a6
Jul 02 21:27:37.630086 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.629984    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:37.683709 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.683682    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:27:37.886054 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.885937    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-jtqwk" event={"ID":"e833578a-9fa9-440d-a13d-197c1043b539","Type":"ContainerStarted","Data":"7a8448c2363fc10c896e4723f4480586f1b8addcfd800a94ce03558f808461a6"}
Jul 02 21:27:37.887188 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:37.887163    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-h7bw7" event={"ID":"6dca3ea7-71e8-4657-a64d-fe18841eac6f","Type":"ContainerStarted","Data":"9628ff88e681d2a7fe4ba402dce624ab0b4ffd31fe463f4fb9cd00ea61af520d"}
Jul 02 21:27:40.903638 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:40.903599    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-jtqwk" event={"ID":"e833578a-9fa9-440d-a13d-197c1043b539","Type":"ContainerStarted","Data":"bced18e2fb3502c90863d8b078e146b992b5b6cfdc963d6acd82314b0587c89c"}
Jul 02 21:27:40.904149 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:40.903668    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-jtqwk"
Jul 02 21:27:40.905151 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:40.905124    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-h7bw7" event={"ID":"6dca3ea7-71e8-4657-a64d-fe18841eac6f","Type":"ContainerStarted","Data":"78917f0e7baf5f6f1d35ee0bda4b75bd1a0841d3dcd18f7f02ec500aa0a1e8d9"}
Jul 02 21:27:40.905277 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:40.905262    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-h7bw7"
Jul 02 21:27:40.921529 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:40.921486    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-jtqwk" podStartSLOduration=2.093735779 podStartE2EDuration="4.921470248s" totalImagesPullingTime="2.827734469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:27:40.919859992 +0000 UTC m=+529.653493325" watchObservedRunningTime="2026-07-02 21:27:40.921470248 +0000 UTC m=+529.655103554"
Jul 02 21:27:40.943432 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:40.943370    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-h7bw7" podStartSLOduration=2.137347563 podStartE2EDuration="4.943354968s" totalImagesPullingTime="2.806007405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:27:40.942108769 +0000 UTC m=+529.675742144" watchObservedRunningTime="2026-07-02 21:27:40.943354968 +0000 UTC m=+529.676988275"
Jul 02 21:27:51.912096 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:51.912019    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-jtqwk"
Jul 02 21:27:51.913835 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:27:51.913816    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-h7bw7"
Jul 02 21:28:14.770883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:28:14.770838    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:12.036594 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:12.036547    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:12.136911 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:12.136875    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:25.464518 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.464481    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:25.773327 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.773246    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:25.856292 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.856249    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:25.906722 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.906681    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"]
Jul 02 21:29:25.910226 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.910199    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:25.912765 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.912728    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 21:29:25.912917 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.912892    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e36cd-predictor-serving-cert\""
Jul 02 21:29:25.914036 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.914013    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q64dw\""
Jul 02 21:29:25.914148 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.914106    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e36cd-kube-rbac-proxy-sar-config\""
Jul 02 21:29:25.914238 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.914222    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 21:29:25.919153 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.918985    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"]
Jul 02 21:29:25.933376 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.933316    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:25.988740 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.988697    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e36cd-predictor-6986f87bbc-4v972\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") " pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:25.988947 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.988773    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b187fc-8887-420e-a767-e1142ff1f0f2-proxy-tls\") pod \"error-404-isvc-e36cd-predictor-6986f87bbc-4v972\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") " pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:25.988947 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:25.988844    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4rg\" (UniqueName: \"kubernetes.io/projected/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-api-access-tn4rg\") pod \"error-404-isvc-e36cd-predictor-6986f87bbc-4v972\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") " pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:26.009829 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.009790    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:26.082847 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.082755    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:26.089542 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.089502    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4rg\" (UniqueName: \"kubernetes.io/projected/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-api-access-tn4rg\") pod \"error-404-isvc-e36cd-predictor-6986f87bbc-4v972\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") " pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:26.089716 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.089566    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e36cd-predictor-6986f87bbc-4v972\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") " pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:26.089716 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.089623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b187fc-8887-420e-a767-e1142ff1f0f2-proxy-tls\") pod \"error-404-isvc-e36cd-predictor-6986f87bbc-4v972\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") " pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:26.089825 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:29:26.089781    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-serving-cert: secret "error-404-isvc-e36cd-predictor-serving-cert" not found
Jul 02 21:29:26.089882 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:29:26.089843    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b187fc-8887-420e-a767-e1142ff1f0f2-proxy-tls podName:00b187fc-8887-420e-a767-e1142ff1f0f2 nodeName:}" failed. No retries permitted until 2026-07-02 21:29:26.589822159 +0000 UTC m=+635.323455450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/00b187fc-8887-420e-a767-e1142ff1f0f2-proxy-tls") pod "error-404-isvc-e36cd-predictor-6986f87bbc-4v972" (UID: "00b187fc-8887-420e-a767-e1142ff1f0f2") : secret "error-404-isvc-e36cd-predictor-serving-cert" not found
Jul 02 21:29:26.090614 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.090586    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e36cd-predictor-6986f87bbc-4v972\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") " pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:26.098945 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.098918    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4rg\" (UniqueName: \"kubernetes.io/projected/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-api-access-tn4rg\") pod \"error-404-isvc-e36cd-predictor-6986f87bbc-4v972\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") " pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:26.155143 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.155107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:26.399964 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.399859    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:26.500186 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.500137    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:29:26.594767 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.594729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b187fc-8887-420e-a767-e1142ff1f0f2-proxy-tls\") pod \"error-404-isvc-e36cd-predictor-6986f87bbc-4v972\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") " pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:26.597233 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.597205    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b187fc-8887-420e-a767-e1142ff1f0f2-proxy-tls\") pod \"error-404-isvc-e36cd-predictor-6986f87bbc-4v972\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") " pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:26.823066 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.823027    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:26.953367 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:26.953342    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"]
Jul 02 21:29:26.956011 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:29:26.955970    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b187fc_8887_420e_a767_e1142ff1f0f2.slice/crio-dd7fff3efd18a080b75117f19ced293f48e0b52674111f2e9c84a55e79e6b213 WatchSource:0}: Error finding container dd7fff3efd18a080b75117f19ced293f48e0b52674111f2e9c84a55e79e6b213: Status 404 returned error can't find the container with id dd7fff3efd18a080b75117f19ced293f48e0b52674111f2e9c84a55e79e6b213
Jul 02 21:29:27.339957 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:27.339918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" event={"ID":"00b187fc-8887-420e-a767-e1142ff1f0f2","Type":"ContainerStarted","Data":"dd7fff3efd18a080b75117f19ced293f48e0b52674111f2e9c84a55e79e6b213"}
Jul 02 21:29:39.411817 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:39.411774    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" event={"ID":"00b187fc-8887-420e-a767-e1142ff1f0f2","Type":"ContainerStarted","Data":"32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121"}
Jul 02 21:29:41.423309 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:41.423273    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" event={"ID":"00b187fc-8887-420e-a767-e1142ff1f0f2","Type":"ContainerStarted","Data":"710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973"}
Jul 02 21:29:41.423885 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:41.423388    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:41.441407 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:41.441354    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" podStartSLOduration=2.598125181 podStartE2EDuration="16.441338201s" totalImagesPullingTime="13.84321302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:29:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:29:41.440421149 +0000 UTC m=+650.174054469" watchObservedRunningTime="2026-07-02 21:29:41.441338201 +0000 UTC m=+650.174971506"
Jul 02 21:29:42.428070 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:42.428035    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:42.429588 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:42.429554    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 21:29:43.432064 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:43.432013    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 21:29:48.436328 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:48.436292    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:29:48.436759 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:48.436733    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 21:29:58.437504 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:29:58.437462    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 21:30:08.437015 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:30:08.436956    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 21:30:18.437252 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:30:18.437206    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 21:30:28.438132 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:30:28.438096    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:30:55.611206 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:30:55.611114    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:30:57.001389 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:30:57.001354    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:09.851570 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:09.851516    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:10.017251 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.017221    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:10.121985 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.121905    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:10.129697 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.129669    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"]
Jul 02 21:31:10.133147 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.133122    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:10.136159 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.136134    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7615e-predictor-serving-cert\""
Jul 02 21:31:10.136269 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.136250    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7615e-kube-rbac-proxy-sar-config\""
Jul 02 21:31:10.144882 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.144859    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"]
Jul 02 21:31:10.184447 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.184410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") " pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:10.184447 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.184460    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp9gw\" (UniqueName: \"kubernetes.io/projected/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-api-access-bp9gw\") pod \"error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") " pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:10.184713 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.184530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b3a01d-fb66-4107-8fec-c45de02b4d92-proxy-tls\") pod \"error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") " pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:10.185690 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.185663    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"]
Jul 02 21:31:10.185954 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.185928    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kserve-container" containerID="cri-o://32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121" gracePeriod=30
Jul 02 21:31:10.186078 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.185959    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kube-rbac-proxy" containerID="cri-o://710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973" gracePeriod=30
Jul 02 21:31:10.206572 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.206545    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:10.267959 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.267920    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:10.285374 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.285349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b3a01d-fb66-4107-8fec-c45de02b4d92-proxy-tls\") pod \"error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") " pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:10.285529 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.285413    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") " pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:10.285529 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.285437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp9gw\" (UniqueName: \"kubernetes.io/projected/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-api-access-bp9gw\") pod \"error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") " pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:10.285529 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:31:10.285451    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-7615e-predictor-serving-cert: secret "error-404-isvc-7615e-predictor-serving-cert" not found
Jul 02 21:31:10.285529 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:31:10.285515    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b3a01d-fb66-4107-8fec-c45de02b4d92-proxy-tls podName:97b3a01d-fb66-4107-8fec-c45de02b4d92 nodeName:}" failed. No retries permitted until 2026-07-02 21:31:10.785500003 +0000 UTC m=+739.519133290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/97b3a01d-fb66-4107-8fec-c45de02b4d92-proxy-tls") pod "error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" (UID: "97b3a01d-fb66-4107-8fec-c45de02b4d92") : secret "error-404-isvc-7615e-predictor-serving-cert" not found
Jul 02 21:31:10.286225 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.286200    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") " pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:10.294454 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.294430    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp9gw\" (UniqueName: \"kubernetes.io/projected/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-api-access-bp9gw\") pod \"error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") " pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:10.369251 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.369206    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:10.789605 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.789552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b3a01d-fb66-4107-8fec-c45de02b4d92-proxy-tls\") pod \"error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") " pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:10.792291 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.792260    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b3a01d-fb66-4107-8fec-c45de02b4d92-proxy-tls\") pod \"error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") " pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:10.807251 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.807217    2564 generic.go:356] "Generic (PLEG): container finished" podID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerID="710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973" exitCode=2
Jul 02 21:31:10.807405 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:10.807291    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" event={"ID":"00b187fc-8887-420e-a767-e1142ff1f0f2","Type":"ContainerDied","Data":"710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973"}
Jul 02 21:31:11.044727 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:11.044636    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:11.180299 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:11.180273    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"]
Jul 02 21:31:11.182013 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:31:11.181972    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b3a01d_fb66_4107_8fec_c45de02b4d92.slice/crio-cb349532bc30561e425c333746a5ca09637d61d2c691c1f2f27ca4b6245cb4a0 WatchSource:0}: Error finding container cb349532bc30561e425c333746a5ca09637d61d2c691c1f2f27ca4b6245cb4a0: Status 404 returned error can't find the container with id cb349532bc30561e425c333746a5ca09637d61d2c691c1f2f27ca4b6245cb4a0
Jul 02 21:31:11.813473 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:11.813433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" event={"ID":"97b3a01d-fb66-4107-8fec-c45de02b4d92","Type":"ContainerStarted","Data":"0c440e5a87511e51a9d526209e2108677c7e13ed6d2b288f6077deaff12bd598"}
Jul 02 21:31:11.813652 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:11.813503    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" event={"ID":"97b3a01d-fb66-4107-8fec-c45de02b4d92","Type":"ContainerStarted","Data":"a22e3cc40d562a5892e4ecc6a604695b5e9e8e4b6ad6609193f98852c79d910e"}
Jul 02 21:31:11.813652 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:11.813530    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" event={"ID":"97b3a01d-fb66-4107-8fec-c45de02b4d92","Type":"ContainerStarted","Data":"cb349532bc30561e425c333746a5ca09637d61d2c691c1f2f27ca4b6245cb4a0"}
Jul 02 21:31:11.813652 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:11.813565    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:11.837589 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:11.837536    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" podStartSLOduration=1.8375207009999999 podStartE2EDuration="1.837520701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:31:11.83603455 +0000 UTC m=+740.569667885" watchObservedRunningTime="2026-07-02 21:31:11.837520701 +0000 UTC m=+740.571154009"
Jul 02 21:31:12.817655 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:12.817619    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:12.818766 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:12.818741    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:31:13.440859 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.440835    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:31:13.512727 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.512694    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-rbac-proxy-sar-config\") pod \"00b187fc-8887-420e-a767-e1142ff1f0f2\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") "
Jul 02 21:31:13.512909 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.512732    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b187fc-8887-420e-a767-e1142ff1f0f2-proxy-tls\") pod \"00b187fc-8887-420e-a767-e1142ff1f0f2\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") "
Jul 02 21:31:13.512909 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.512773    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn4rg\" (UniqueName: \"kubernetes.io/projected/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-api-access-tn4rg\") pod \"00b187fc-8887-420e-a767-e1142ff1f0f2\" (UID: \"00b187fc-8887-420e-a767-e1142ff1f0f2\") "
Jul 02 21:31:13.513154 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.513125    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00b187fc-8887-420e-a767-e1142ff1f0f2" (UID: "00b187fc-8887-420e-a767-e1142ff1f0f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:31:13.514963 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.514939    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-api-access-tn4rg" (OuterVolumeSpecName: "kube-api-access-tn4rg") pod "00b187fc-8887-420e-a767-e1142ff1f0f2" (UID: "00b187fc-8887-420e-a767-e1142ff1f0f2"). InnerVolumeSpecName "kube-api-access-tn4rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:31:13.515068 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.515045    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b187fc-8887-420e-a767-e1142ff1f0f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00b187fc-8887-420e-a767-e1142ff1f0f2" (UID: "00b187fc-8887-420e-a767-e1142ff1f0f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:31:13.613956 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.613864    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:31:13.613956 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.613895    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b187fc-8887-420e-a767-e1142ff1f0f2-proxy-tls\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:31:13.613956 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.613905    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4rg\" (UniqueName: \"kubernetes.io/projected/00b187fc-8887-420e-a767-e1142ff1f0f2-kube-api-access-tn4rg\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:31:13.823152 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.823115    2564 generic.go:356] "Generic (PLEG): container finished" podID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerID="32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121" exitCode=0
Jul 02 21:31:13.823572 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.823169    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"
Jul 02 21:31:13.823572 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.823193    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" event={"ID":"00b187fc-8887-420e-a767-e1142ff1f0f2","Type":"ContainerDied","Data":"32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121"}
Jul 02 21:31:13.823572 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.823247    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" event={"ID":"00b187fc-8887-420e-a767-e1142ff1f0f2","Type":"ContainerDied","Data":"dd7fff3efd18a080b75117f19ced293f48e0b52674111f2e9c84a55e79e6b213"}
Jul 02 21:31:13.823572 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.823273    2564 scope.go:117] "RemoveContainer" containerID="710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973"
Jul 02 21:31:13.823873 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.823829    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:31:13.831978 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.831961    2564 scope.go:117] "RemoveContainer" containerID="32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121"
Jul 02 21:31:13.840221 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.840199    2564 scope.go:117] "RemoveContainer" containerID="710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973"
Jul 02 21:31:13.840483 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:31:13.840462    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973\": container with ID starting with 710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973 not found: ID does not exist" containerID="710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973"
Jul 02 21:31:13.840559 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.840491    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973"} err="failed to get container status \"710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973\": rpc error: code = NotFound desc = could not find container \"710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973\": container with ID starting with 710a9d71546b0a89f1fba4a56a1ee160c024a192cc346bc6efce9c52c22ea973 not found: ID does not exist"
Jul 02 21:31:13.840559 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.840510    2564 scope.go:117] "RemoveContainer" containerID="32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121"
Jul 02 21:31:13.840730 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:31:13.840710    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121\": container with ID starting with 32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121 not found: ID does not exist" containerID="32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121"
Jul 02 21:31:13.840787 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.840740    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121"} err="failed to get container status \"32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121\": rpc error: code = NotFound desc = could not find container \"32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121\": container with ID starting with 32b8cb80340009e7fe44a094dd9d8e707caf617a773e53f649ec9cdfaca74121 not found: ID does not exist"
Jul 02 21:31:13.845469 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.845449    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"]
Jul 02 21:31:13.849750 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:13.849722    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972"]
Jul 02 21:31:14.432504 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:14.432441    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e36cd-predictor-6986f87bbc-4v972" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": context deadline exceeded"
Jul 02 21:31:15.833796 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:15.833760    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" path="/var/lib/kubelet/pods/00b187fc-8887-420e-a767-e1142ff1f0f2/volumes"
Jul 02 21:31:18.828314 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:18.828279    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:31:18.829155 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:18.829112    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:31:25.580558 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:25.580516    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:26.485802 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:26.485768    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:28.829756 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:28.829709    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:31:35.698362 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.698313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:35.842289 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.842211    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:35.924459 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.920344    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:35.933589 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.933554    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"]
Jul 02 21:31:35.934055 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.934036    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kserve-container"
Jul 02 21:31:35.934055 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.934054    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kserve-container"
Jul 02 21:31:35.934235 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.934080    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kube-rbac-proxy"
Jul 02 21:31:35.934235 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.934089    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kube-rbac-proxy"
Jul 02 21:31:35.934235 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.934167    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kserve-container"
Jul 02 21:31:35.934235 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.934178    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b187fc-8887-420e-a767-e1142ff1f0f2" containerName="kube-rbac-proxy"
Jul 02 21:31:35.938824 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.938801    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:35.941732 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.941703    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a3b0c-kube-rbac-proxy-sar-config\""
Jul 02 21:31:35.941888 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.941734    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a3b0c-predictor-serving-cert\""
Jul 02 21:31:35.948706 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:35.948647    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"]
Jul 02 21:31:36.024683 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.024260    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:36.099729 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.099680    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:36.106921 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.106894    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbkqf\" (UniqueName: \"kubernetes.io/projected/4518a8cb-7f8f-4421-b588-92c52963712b-kube-api-access-pbkqf\") pod \"error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") " pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:36.107086 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.106935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4518a8cb-7f8f-4421-b588-92c52963712b-proxy-tls\") pod \"error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") " pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:36.107086 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.107046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4518a8cb-7f8f-4421-b588-92c52963712b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") " pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:36.207688 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.207590    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4518a8cb-7f8f-4421-b588-92c52963712b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") " pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:36.207688 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.207636    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbkqf\" (UniqueName: \"kubernetes.io/projected/4518a8cb-7f8f-4421-b588-92c52963712b-kube-api-access-pbkqf\") pod \"error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") " pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:36.207688 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.207672    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4518a8cb-7f8f-4421-b588-92c52963712b-proxy-tls\") pod \"error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") " pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:36.207982 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:31:36.207816    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-serving-cert: secret "error-404-isvc-a3b0c-predictor-serving-cert" not found
Jul 02 21:31:36.207982 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:31:36.207894    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4518a8cb-7f8f-4421-b588-92c52963712b-proxy-tls podName:4518a8cb-7f8f-4421-b588-92c52963712b nodeName:}" failed. No retries permitted until 2026-07-02 21:31:36.70787334 +0000 UTC m=+765.441506640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4518a8cb-7f8f-4421-b588-92c52963712b-proxy-tls") pod "error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" (UID: "4518a8cb-7f8f-4421-b588-92c52963712b") : secret "error-404-isvc-a3b0c-predictor-serving-cert" not found
Jul 02 21:31:36.208387 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.208362    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4518a8cb-7f8f-4421-b588-92c52963712b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") " pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:36.217533 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.217499    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbkqf\" (UniqueName: \"kubernetes.io/projected/4518a8cb-7f8f-4421-b588-92c52963712b-kube-api-access-pbkqf\") pod \"error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") " pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:36.302858 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.302811    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:36.426355 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.426323    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:31:36.712532 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.712500    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4518a8cb-7f8f-4421-b588-92c52963712b-proxy-tls\") pod \"error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") " pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:36.714912 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.714889    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4518a8cb-7f8f-4421-b588-92c52963712b-proxy-tls\") pod \"error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") " pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:36.853047 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.852983    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:36.987273 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:36.987242    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"]
Jul 02 21:31:36.989491 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:31:36.989457    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4518a8cb_7f8f_4421_b588_92c52963712b.slice/crio-f91a79df73ecb59b1ad17139c98556d561e5b6f1aaff4b8eb6d5adcd0bbdd708 WatchSource:0}: Error finding container f91a79df73ecb59b1ad17139c98556d561e5b6f1aaff4b8eb6d5adcd0bbdd708: Status 404 returned error can't find the container with id f91a79df73ecb59b1ad17139c98556d561e5b6f1aaff4b8eb6d5adcd0bbdd708
Jul 02 21:31:37.929657 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:37.929619    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" event={"ID":"4518a8cb-7f8f-4421-b588-92c52963712b","Type":"ContainerStarted","Data":"6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d"}
Jul 02 21:31:37.930123 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:37.929679    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" event={"ID":"4518a8cb-7f8f-4421-b588-92c52963712b","Type":"ContainerStarted","Data":"688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1"}
Jul 02 21:31:37.930123 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:37.929702    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" event={"ID":"4518a8cb-7f8f-4421-b588-92c52963712b","Type":"ContainerStarted","Data":"f91a79df73ecb59b1ad17139c98556d561e5b6f1aaff4b8eb6d5adcd0bbdd708"}
Jul 02 21:31:37.930123 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:37.929768    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:37.948440 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:37.948393    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" podStartSLOduration=2.948378016 podStartE2EDuration="2.948378016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:31:37.947486415 +0000 UTC m=+766.681119736" watchObservedRunningTime="2026-07-02 21:31:37.948378016 +0000 UTC m=+766.682011324"
Jul 02 21:31:38.829141 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:38.829042    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:31:38.935303 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:38.935270    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:38.936807 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:38.936778    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 21:31:39.939361 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:39.939313    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 21:31:44.944184 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:44.944150    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:31:44.944679 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:44.944643    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 21:31:48.829031 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:48.828979    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 21:31:54.945527 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:54.945488    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 21:31:58.830052 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:31:58.830022    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:32:04.945647 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:32:04.945605    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 21:32:06.786686 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:32:06.786634    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:06.841349 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:32:06.841311    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:10.061544 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:32:10.061496    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:10.172757 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:32:10.172713    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:14.945098 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:32:14.945013    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 21:32:24.946167 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:32:24.946127    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:32:35.880197 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:32:35.880151    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:36.414166 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:32:36.414122    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:32:43.807902 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:32:43.807869    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:34:47.580118 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:34:47.580074    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:35:41.140419 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:35:41.140378    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:36:44.928103 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:36:44.928003    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:24.676974 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.676923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:24.797486 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.797457    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:24.900039 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.899983    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:24.925803 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.925775    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"]
Jul 02 21:40:24.929115 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.929061    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:24.931725 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.931708    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ee70d-predictor-serving-cert\""
Jul 02 21:40:24.931843 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.931735    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ee70d-kube-rbac-proxy-sar-config\""
Jul 02 21:40:24.941722 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.941701    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"]
Jul 02 21:40:24.970253 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.970220    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"]
Jul 02 21:40:24.970474 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.970443    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kserve-container" containerID="cri-o://a22e3cc40d562a5892e4ecc6a604695b5e9e8e4b6ad6609193f98852c79d910e" gracePeriod=30
Jul 02 21:40:24.970599 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.970484    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kube-rbac-proxy" containerID="cri-o://0c440e5a87511e51a9d526209e2108677c7e13ed6d2b288f6077deaff12bd598" gracePeriod=30
Jul 02 21:40:24.972982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:24.972954    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:25.034462 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.034417    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9624fa5-352c-4268-b4d6-a1668d4aa00f-proxy-tls\") pod \"error-404-isvc-ee70d-predictor-5797bfd779-l6tvl\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") " pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:25.034620 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.034537    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bq4k\" (UniqueName: \"kubernetes.io/projected/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-api-access-5bq4k\") pod \"error-404-isvc-ee70d-predictor-5797bfd779-l6tvl\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") " pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:25.034691 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.034630    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ee70d-predictor-5797bfd779-l6tvl\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") " pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:25.035072 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.035026    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:25.135691 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.135656    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ee70d-predictor-5797bfd779-l6tvl\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") " pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:25.135862 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.135695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9624fa5-352c-4268-b4d6-a1668d4aa00f-proxy-tls\") pod \"error-404-isvc-ee70d-predictor-5797bfd779-l6tvl\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") " pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:25.135862 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.135740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bq4k\" (UniqueName: \"kubernetes.io/projected/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-api-access-5bq4k\") pod \"error-404-isvc-ee70d-predictor-5797bfd779-l6tvl\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") " pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:25.135985 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:40:25.135857    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-serving-cert: secret "error-404-isvc-ee70d-predictor-serving-cert" not found
Jul 02 21:40:25.135985 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:40:25.135921    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9624fa5-352c-4268-b4d6-a1668d4aa00f-proxy-tls podName:c9624fa5-352c-4268-b4d6-a1668d4aa00f nodeName:}" failed. No retries permitted until 2026-07-02 21:40:25.635902831 +0000 UTC m=+1294.369536121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c9624fa5-352c-4268-b4d6-a1668d4aa00f-proxy-tls") pod "error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" (UID: "c9624fa5-352c-4268-b4d6-a1668d4aa00f") : secret "error-404-isvc-ee70d-predictor-serving-cert" not found
Jul 02 21:40:25.136308 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.136287    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ee70d-predictor-5797bfd779-l6tvl\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") " pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:25.144906 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.144877    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bq4k\" (UniqueName: \"kubernetes.io/projected/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-api-access-5bq4k\") pod \"error-404-isvc-ee70d-predictor-5797bfd779-l6tvl\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") " pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:25.162171 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.162135    2564 generic.go:356] "Generic (PLEG): container finished" podID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerID="0c440e5a87511e51a9d526209e2108677c7e13ed6d2b288f6077deaff12bd598" exitCode=2
Jul 02 21:40:25.162290 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.162212    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" event={"ID":"97b3a01d-fb66-4107-8fec-c45de02b4d92","Type":"ContainerDied","Data":"0c440e5a87511e51a9d526209e2108677c7e13ed6d2b288f6077deaff12bd598"}
Jul 02 21:40:25.286135 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.286098    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:25.640085 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.639978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9624fa5-352c-4268-b4d6-a1668d4aa00f-proxy-tls\") pod \"error-404-isvc-ee70d-predictor-5797bfd779-l6tvl\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") " pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:25.642407 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.642374    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9624fa5-352c-4268-b4d6-a1668d4aa00f-proxy-tls\") pod \"error-404-isvc-ee70d-predictor-5797bfd779-l6tvl\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") " pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:25.839010 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.838975    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:25.972308 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:25.972281    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"]
Jul 02 21:40:25.974624 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:40:25.974597    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9624fa5_352c_4268_b4d6_a1668d4aa00f.slice/crio-17446a81ebe80a93ba48428386d4fa4232bb4fa29d2f0359a804787814a6bdee WatchSource:0}: Error finding container 17446a81ebe80a93ba48428386d4fa4232bb4fa29d2f0359a804787814a6bdee: Status 404 returned error can't find the container with id 17446a81ebe80a93ba48428386d4fa4232bb4fa29d2f0359a804787814a6bdee
Jul 02 21:40:26.168455 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:26.168419    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" event={"ID":"c9624fa5-352c-4268-b4d6-a1668d4aa00f","Type":"ContainerStarted","Data":"6207f71468a2a96c92b70ebc788a4d8a70b5f19ba3335a4a553c8186d8d20db4"}
Jul 02 21:40:26.168631 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:26.168474    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" event={"ID":"c9624fa5-352c-4268-b4d6-a1668d4aa00f","Type":"ContainerStarted","Data":"c91a4181270fe6e3951d362fe783b050b9e5fc5b4be758fbd942ba2af50dfdde"}
Jul 02 21:40:26.168631 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:26.168494    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" event={"ID":"c9624fa5-352c-4268-b4d6-a1668d4aa00f","Type":"ContainerStarted","Data":"17446a81ebe80a93ba48428386d4fa4232bb4fa29d2f0359a804787814a6bdee"}
Jul 02 21:40:26.168745 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:26.168640    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:26.168804 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:26.168790    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:26.170275 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:26.170245    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 21:40:26.188653 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:26.188542    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" podStartSLOduration=2.188521848 podStartE2EDuration="2.188521848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:40:26.188065001 +0000 UTC m=+1294.921698330" watchObservedRunningTime="2026-07-02 21:40:26.188521848 +0000 UTC m=+1294.922155157"
Jul 02 21:40:27.172764 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:27.172718    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 21:40:28.178604 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.178574    2564 generic.go:356] "Generic (PLEG): container finished" podID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerID="a22e3cc40d562a5892e4ecc6a604695b5e9e8e4b6ad6609193f98852c79d910e" exitCode=0
Jul 02 21:40:28.178939 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.178640    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" event={"ID":"97b3a01d-fb66-4107-8fec-c45de02b4d92","Type":"ContainerDied","Data":"a22e3cc40d562a5892e4ecc6a604695b5e9e8e4b6ad6609193f98852c79d910e"}
Jul 02 21:40:28.233091 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.233067    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:40:28.363005 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.362911    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b3a01d-fb66-4107-8fec-c45de02b4d92-proxy-tls\") pod \"97b3a01d-fb66-4107-8fec-c45de02b4d92\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") "
Jul 02 21:40:28.363005 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.362973    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp9gw\" (UniqueName: \"kubernetes.io/projected/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-api-access-bp9gw\") pod \"97b3a01d-fb66-4107-8fec-c45de02b4d92\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") "
Jul 02 21:40:28.363194 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.363011    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-rbac-proxy-sar-config\") pod \"97b3a01d-fb66-4107-8fec-c45de02b4d92\" (UID: \"97b3a01d-fb66-4107-8fec-c45de02b4d92\") "
Jul 02 21:40:28.363419 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.363394    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97b3a01d-fb66-4107-8fec-c45de02b4d92" (UID: "97b3a01d-fb66-4107-8fec-c45de02b4d92"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:40:28.365049 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.365023    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-api-access-bp9gw" (OuterVolumeSpecName: "kube-api-access-bp9gw") pod "97b3a01d-fb66-4107-8fec-c45de02b4d92" (UID: "97b3a01d-fb66-4107-8fec-c45de02b4d92"). InnerVolumeSpecName "kube-api-access-bp9gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:40:28.365104 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.365027    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b3a01d-fb66-4107-8fec-c45de02b4d92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97b3a01d-fb66-4107-8fec-c45de02b4d92" (UID: "97b3a01d-fb66-4107-8fec-c45de02b4d92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:40:28.463511 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.463471    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b3a01d-fb66-4107-8fec-c45de02b4d92-proxy-tls\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:40:28.463511 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.463506    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp9gw\" (UniqueName: \"kubernetes.io/projected/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-api-access-bp9gw\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:40:28.463511 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:28.463517    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b3a01d-fb66-4107-8fec-c45de02b4d92-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:40:29.184560 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:29.184525    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"
Jul 02 21:40:29.184560 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:29.184541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj" event={"ID":"97b3a01d-fb66-4107-8fec-c45de02b4d92","Type":"ContainerDied","Data":"cb349532bc30561e425c333746a5ca09637d61d2c691c1f2f27ca4b6245cb4a0"}
Jul 02 21:40:29.185172 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:29.184602    2564 scope.go:117] "RemoveContainer" containerID="0c440e5a87511e51a9d526209e2108677c7e13ed6d2b288f6077deaff12bd598"
Jul 02 21:40:29.193635 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:29.193618    2564 scope.go:117] "RemoveContainer" containerID="a22e3cc40d562a5892e4ecc6a604695b5e9e8e4b6ad6609193f98852c79d910e"
Jul 02 21:40:29.206856 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:29.206832    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"]
Jul 02 21:40:29.209169 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:29.209148    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7615e-predictor-5bf56c7b5c-fxmsj"]
Jul 02 21:40:29.833806 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:29.833769    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" path="/var/lib/kubelet/pods/97b3a01d-fb66-4107-8fec-c45de02b4d92/volumes"
Jul 02 21:40:32.177867 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:32.177839    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:40:32.178410 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:32.178387    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 21:40:42.179300 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:42.179260    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 21:40:50.655734 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.655696    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:50.761365 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.761327    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:50.863872 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.863835    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:50.894064 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.894023    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"]
Jul 02 21:40:50.894521 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.894494    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kserve-container"
Jul 02 21:40:50.894687 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.894664    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kserve-container"
Jul 02 21:40:50.894822 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.894695    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kube-rbac-proxy"
Jul 02 21:40:50.894822 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.894704    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kube-rbac-proxy"
Jul 02 21:40:50.894927 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.894830    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kserve-container"
Jul 02 21:40:50.894927 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.894841    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b3a01d-fb66-4107-8fec-c45de02b4d92" containerName="kube-rbac-proxy"
Jul 02 21:40:50.898055 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.898032    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:50.901086 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.901060    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1702f-predictor-serving-cert\""
Jul 02 21:40:50.901191 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.901060    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1702f-kube-rbac-proxy-sar-config\""
Jul 02 21:40:50.907898 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.907872    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"]
Jul 02 21:40:50.935538 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.935506    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"]
Jul 02 21:40:50.935836 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.935803    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kserve-container" containerID="cri-o://688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1" gracePeriod=30
Jul 02 21:40:50.935974 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.935915    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kube-rbac-proxy" containerID="cri-o://6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d" gracePeriod=30
Jul 02 21:40:50.937704 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:50.937495    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:51.000304 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.000248    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:51.053981 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.053957    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0530481-196f-444c-ae68-287e6d9b4138-proxy-tls\") pod \"error-404-isvc-1702f-predictor-786664c4bf-btbvk\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") " pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:51.054110 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.054024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkm4l\" (UniqueName: \"kubernetes.io/projected/c0530481-196f-444c-ae68-287e6d9b4138-kube-api-access-wkm4l\") pod \"error-404-isvc-1702f-predictor-786664c4bf-btbvk\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") " pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:51.054110 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.054077    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0530481-196f-444c-ae68-287e6d9b4138-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1702f-predictor-786664c4bf-btbvk\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") " pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:51.101609 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.101555    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:51.155334 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.155300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0530481-196f-444c-ae68-287e6d9b4138-proxy-tls\") pod \"error-404-isvc-1702f-predictor-786664c4bf-btbvk\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") " pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:51.155531 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.155349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkm4l\" (UniqueName: \"kubernetes.io/projected/c0530481-196f-444c-ae68-287e6d9b4138-kube-api-access-wkm4l\") pod \"error-404-isvc-1702f-predictor-786664c4bf-btbvk\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") " pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:51.155531 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.155389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0530481-196f-444c-ae68-287e6d9b4138-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1702f-predictor-786664c4bf-btbvk\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") " pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:51.155531 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:40:51.155480    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1702f-predictor-serving-cert: secret "error-404-isvc-1702f-predictor-serving-cert" not found
Jul 02 21:40:51.155683 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:40:51.155564    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0530481-196f-444c-ae68-287e6d9b4138-proxy-tls podName:c0530481-196f-444c-ae68-287e6d9b4138 nodeName:}" failed. No retries permitted until 2026-07-02 21:40:51.655530966 +0000 UTC m=+1320.389164252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c0530481-196f-444c-ae68-287e6d9b4138-proxy-tls") pod "error-404-isvc-1702f-predictor-786664c4bf-btbvk" (UID: "c0530481-196f-444c-ae68-287e6d9b4138") : secret "error-404-isvc-1702f-predictor-serving-cert" not found
Jul 02 21:40:51.156071 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.156054    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0530481-196f-444c-ae68-287e6d9b4138-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1702f-predictor-786664c4bf-btbvk\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") " pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:51.164477 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.164416    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkm4l\" (UniqueName: \"kubernetes.io/projected/c0530481-196f-444c-ae68-287e6d9b4138-kube-api-access-wkm4l\") pod \"error-404-isvc-1702f-predictor-786664c4bf-btbvk\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") " pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:51.286094 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.286048    2564 generic.go:356] "Generic (PLEG): container finished" podID="4518a8cb-7f8f-4421-b588-92c52963712b" containerID="6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d" exitCode=2
Jul 02 21:40:51.286259 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.286112    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" event={"ID":"4518a8cb-7f8f-4421-b588-92c52963712b","Type":"ContainerDied","Data":"6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d"}
Jul 02 21:40:51.290711 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.290687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:51.658420 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.658384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0530481-196f-444c-ae68-287e6d9b4138-proxy-tls\") pod \"error-404-isvc-1702f-predictor-786664c4bf-btbvk\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") " pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:51.660814 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.660787    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0530481-196f-444c-ae68-287e6d9b4138-proxy-tls\") pod \"error-404-isvc-1702f-predictor-786664c4bf-btbvk\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") " pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:51.811498 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.811468    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:51.953235 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:51.953151    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"]
Jul 02 21:40:51.955663 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:40:51.955637    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0530481_196f_444c_ae68_287e6d9b4138.slice/crio-0a14c20d0bb08e720f903037583daf6236ca1850b80df19118e89627342fd568 WatchSource:0}: Error finding container 0a14c20d0bb08e720f903037583daf6236ca1850b80df19118e89627342fd568: Status 404 returned error can't find the container with id 0a14c20d0bb08e720f903037583daf6236ca1850b80df19118e89627342fd568
Jul 02 21:40:52.179065 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:52.179016    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 21:40:52.294105 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:52.294006    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" event={"ID":"c0530481-196f-444c-ae68-287e6d9b4138","Type":"ContainerStarted","Data":"a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f"}
Jul 02 21:40:52.294105 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:52.294081    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:52.294324 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:52.294107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" event={"ID":"c0530481-196f-444c-ae68-287e6d9b4138","Type":"ContainerStarted","Data":"115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4"}
Jul 02 21:40:52.294324 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:52.294128    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:52.294324 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:52.294156    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" event={"ID":"c0530481-196f-444c-ae68-287e6d9b4138","Type":"ContainerStarted","Data":"0a14c20d0bb08e720f903037583daf6236ca1850b80df19118e89627342fd568"}
Jul 02 21:40:52.295604 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:52.295574    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:40:52.312464 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:52.312415    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" podStartSLOduration=2.312400743 podStartE2EDuration="2.312400743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:40:52.310613875 +0000 UTC m=+1321.044247212" watchObservedRunningTime="2026-07-02 21:40:52.312400743 +0000 UTC m=+1321.046034050"
Jul 02 21:40:53.041512 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:53.041480    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:40:53.298620 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:53.298530    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:40:54.940401 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:54.940356    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 21:40:54.944702 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:54.944669    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 21:40:57.496045 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:57.496019    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:40:57.610310 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:57.610222    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4518a8cb-7f8f-4421-b588-92c52963712b-kube-rbac-proxy-sar-config\") pod \"4518a8cb-7f8f-4421-b588-92c52963712b\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") "
Jul 02 21:40:57.610310 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:57.610265    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4518a8cb-7f8f-4421-b588-92c52963712b-proxy-tls\") pod \"4518a8cb-7f8f-4421-b588-92c52963712b\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") "
Jul 02 21:40:57.610500 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:57.610315    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbkqf\" (UniqueName: \"kubernetes.io/projected/4518a8cb-7f8f-4421-b588-92c52963712b-kube-api-access-pbkqf\") pod \"4518a8cb-7f8f-4421-b588-92c52963712b\" (UID: \"4518a8cb-7f8f-4421-b588-92c52963712b\") "
Jul 02 21:40:57.610562 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:57.610540    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4518a8cb-7f8f-4421-b588-92c52963712b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4518a8cb-7f8f-4421-b588-92c52963712b" (UID: "4518a8cb-7f8f-4421-b588-92c52963712b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:40:57.612388 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:57.612356    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4518a8cb-7f8f-4421-b588-92c52963712b-kube-api-access-pbkqf" (OuterVolumeSpecName: "kube-api-access-pbkqf") pod "4518a8cb-7f8f-4421-b588-92c52963712b" (UID: "4518a8cb-7f8f-4421-b588-92c52963712b"). InnerVolumeSpecName "kube-api-access-pbkqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:40:57.612516 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:57.612394    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4518a8cb-7f8f-4421-b588-92c52963712b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4518a8cb-7f8f-4421-b588-92c52963712b" (UID: "4518a8cb-7f8f-4421-b588-92c52963712b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:40:57.710960 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:57.710920    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbkqf\" (UniqueName: \"kubernetes.io/projected/4518a8cb-7f8f-4421-b588-92c52963712b-kube-api-access-pbkqf\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:40:57.710960 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:57.710954    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4518a8cb-7f8f-4421-b588-92c52963712b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:40:57.710960 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:57.710964    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4518a8cb-7f8f-4421-b588-92c52963712b-proxy-tls\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:40:58.303592 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.303563    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:40:58.304047 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.304021    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:40:58.321709 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.321678    2564 generic.go:356] "Generic (PLEG): container finished" podID="4518a8cb-7f8f-4421-b588-92c52963712b" containerID="688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1" exitCode=0
Jul 02 21:40:58.321893 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.321728    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" event={"ID":"4518a8cb-7f8f-4421-b588-92c52963712b","Type":"ContainerDied","Data":"688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1"}
Jul 02 21:40:58.322031 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.321743    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"
Jul 02 21:40:58.322164 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.322052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm" event={"ID":"4518a8cb-7f8f-4421-b588-92c52963712b","Type":"ContainerDied","Data":"f91a79df73ecb59b1ad17139c98556d561e5b6f1aaff4b8eb6d5adcd0bbdd708"}
Jul 02 21:40:58.322164 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.322096    2564 scope.go:117] "RemoveContainer" containerID="6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d"
Jul 02 21:40:58.330684 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.330646    2564 scope.go:117] "RemoveContainer" containerID="688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1"
Jul 02 21:40:58.338153 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.338126    2564 scope.go:117] "RemoveContainer" containerID="6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d"
Jul 02 21:40:58.338421 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:40:58.338403    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d\": container with ID starting with 6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d not found: ID does not exist" containerID="6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d"
Jul 02 21:40:58.338480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.338429    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d"} err="failed to get container status \"6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d\": rpc error: code = NotFound desc = could not find container \"6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d\": container with ID starting with 6e293dc8abe4899f01dad0c12c1b8bea0bb8c10a5bbbef998b0f619781cb466d not found: ID does not exist"
Jul 02 21:40:58.338480 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.338446    2564 scope.go:117] "RemoveContainer" containerID="688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1"
Jul 02 21:40:58.338669 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:40:58.338651    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1\": container with ID starting with 688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1 not found: ID does not exist" containerID="688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1"
Jul 02 21:40:58.338761 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.338680    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1"} err="failed to get container status \"688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1\": rpc error: code = NotFound desc = could not find container \"688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1\": container with ID starting with 688791b2ba076c0e43fabb3b50fb619f2ce6712a0bc19cdf574783d68de0f2c1 not found: ID does not exist"
Jul 02 21:40:58.339429 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.339411    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"]
Jul 02 21:40:58.342599 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:58.342580    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3b0c-predictor-5dd5b5fb79-sl8qm"]
Jul 02 21:40:59.833777 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:40:59.833741    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" path="/var/lib/kubelet/pods/4518a8cb-7f8f-4421-b588-92c52963712b/volumes"
Jul 02 21:41:02.178526 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:02.178483    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 21:41:08.304401 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:08.304362    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:41:12.179184 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:12.179153    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:41:18.304646 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:18.304561    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:41:24.876100 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:24.876068    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:25.106134 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:25.106100    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:28.304214 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:28.304171    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 21:41:34.960496 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:34.960463    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:35.016749 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.016711    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"]
Jul 02 21:41:35.017013 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.016966    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kserve-container" containerID="cri-o://c91a4181270fe6e3951d362fe783b050b9e5fc5b4be758fbd942ba2af50dfdde" gracePeriod=30
Jul 02 21:41:35.017098 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.017021    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kube-rbac-proxy" containerID="cri-o://6207f71468a2a96c92b70ebc788a4d8a70b5f19ba3335a4a553c8186d8d20db4" gracePeriod=30
Jul 02 21:41:35.062904 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.062852    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:35.156140 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.156102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:35.228917 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.228837    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"]
Jul 02 21:41:35.229198 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.229184    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kube-rbac-proxy"
Jul 02 21:41:35.229244 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.229200    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kube-rbac-proxy"
Jul 02 21:41:35.229244 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.229211    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kserve-container"
Jul 02 21:41:35.229244 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.229217    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kserve-container"
Jul 02 21:41:35.229347 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.229276    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kube-rbac-proxy"
Jul 02 21:41:35.229347 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.229286    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4518a8cb-7f8f-4421-b588-92c52963712b" containerName="kserve-container"
Jul 02 21:41:35.232363 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.232342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:35.235109 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:41:35.235082    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr\" is forbidden: User \"system:node:ip-10-0-130-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-130-121.ec2.internal' and this object" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:35.235521 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:41:35.235494    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"error-404-isvc-87dcc-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-130-121.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-130-121.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-87dcc-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Jul 02 21:41:35.235714 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:41:35.235697    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"error-404-isvc-87dcc-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-130-121.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-130-121.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-87dcc-predictor-serving-cert\"" type="*v1.Secret"
Jul 02 21:41:35.249562 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.249538    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"]
Jul 02 21:41:35.257819 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.257794    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:35.300626 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.300594    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbr4t\" (UniqueName: \"kubernetes.io/projected/cb66a45d-e080-4406-9218-0c4bf5107548-kube-api-access-rbr4t\") pod \"error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") " pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:35.300794 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.300641    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb66a45d-e080-4406-9218-0c4bf5107548-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") " pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:35.300794 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.300714    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb66a45d-e080-4406-9218-0c4bf5107548-proxy-tls\") pod \"error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") " pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:35.362106 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.362073    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:35.401300 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.401268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb66a45d-e080-4406-9218-0c4bf5107548-proxy-tls\") pod \"error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") " pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:35.401448 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.401321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbr4t\" (UniqueName: \"kubernetes.io/projected/cb66a45d-e080-4406-9218-0c4bf5107548-kube-api-access-rbr4t\") pod \"error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") " pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:35.401448 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.401360    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb66a45d-e080-4406-9218-0c4bf5107548-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") " pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:35.409953 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.409929    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbr4t\" (UniqueName: \"kubernetes.io/projected/cb66a45d-e080-4406-9218-0c4bf5107548-kube-api-access-rbr4t\") pod \"error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") " pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:35.482699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.482621    2564 generic.go:356] "Generic (PLEG): container finished" podID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerID="6207f71468a2a96c92b70ebc788a4d8a70b5f19ba3335a4a553c8186d8d20db4" exitCode=2
Jul 02 21:41:35.482699 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:35.482657    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" event={"ID":"c9624fa5-352c-4268-b4d6-a1668d4aa00f","Type":"ContainerDied","Data":"6207f71468a2a96c92b70ebc788a4d8a70b5f19ba3335a4a553c8186d8d20db4"}
Jul 02 21:41:36.132773 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:36.132743    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-87dcc-kube-rbac-proxy-sar-config\""
Jul 02 21:41:36.142901 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:36.142875    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb66a45d-e080-4406-9218-0c4bf5107548-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") " pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:36.402125 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:41:36.402034    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-serving-cert: failed to sync secret cache: timed out waiting for the condition
Jul 02 21:41:36.402125 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:41:36.402129    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb66a45d-e080-4406-9218-0c4bf5107548-proxy-tls podName:cb66a45d-e080-4406-9218-0c4bf5107548 nodeName:}" failed. No retries permitted until 2026-07-02 21:41:36.902111317 +0000 UTC m=+1365.635744603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cb66a45d-e080-4406-9218-0c4bf5107548-proxy-tls") pod "error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" (UID: "cb66a45d-e080-4406-9218-0c4bf5107548") : failed to sync secret cache: timed out waiting for the condition
Jul 02 21:41:36.757530 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:36.757498    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-87dcc-predictor-serving-cert\""
Jul 02 21:41:36.914359 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:36.914312    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb66a45d-e080-4406-9218-0c4bf5107548-proxy-tls\") pod \"error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") " pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:36.916683 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:36.916651    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb66a45d-e080-4406-9218-0c4bf5107548-proxy-tls\") pod \"error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") " pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:37.042227 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:37.042128    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:37.168067 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:37.168041    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"]
Jul 02 21:41:37.170374 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:41:37.170343    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb66a45d_e080_4406_9218_0c4bf5107548.slice/crio-96f2f4a4cff7c6038bf304c691cd3c74b6e7012164d54d135d8df19a032e51e9 WatchSource:0}: Error finding container 96f2f4a4cff7c6038bf304c691cd3c74b6e7012164d54d135d8df19a032e51e9: Status 404 returned error can't find the container with id 96f2f4a4cff7c6038bf304c691cd3c74b6e7012164d54d135d8df19a032e51e9
Jul 02 21:41:37.173056 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:37.173029    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 02 21:41:37.495283 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:37.495246    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" event={"ID":"cb66a45d-e080-4406-9218-0c4bf5107548","Type":"ContainerStarted","Data":"c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636"}
Jul 02 21:41:37.495487 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:37.495409    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:37.495487 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:37.495449    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" event={"ID":"cb66a45d-e080-4406-9218-0c4bf5107548","Type":"ContainerStarted","Data":"a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e"}
Jul 02 21:41:37.495487 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:37.495472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" event={"ID":"cb66a45d-e080-4406-9218-0c4bf5107548","Type":"ContainerStarted","Data":"96f2f4a4cff7c6038bf304c691cd3c74b6e7012164d54d135d8df19a032e51e9"}
Jul 02 21:41:37.516690 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:37.516645    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" podStartSLOduration=2.516630523 podStartE2EDuration="2.516630523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:41:37.514423433 +0000 UTC m=+1366.248056753" watchObservedRunningTime="2026-07-02 21:41:37.516630523 +0000 UTC m=+1366.250263828"
Jul 02 21:41:38.305669 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.305638    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:41:38.504016 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.503962    2564 generic.go:356] "Generic (PLEG): container finished" podID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerID="c91a4181270fe6e3951d362fe783b050b9e5fc5b4be758fbd942ba2af50dfdde" exitCode=0
Jul 02 21:41:38.504156 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.504077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" event={"ID":"c9624fa5-352c-4268-b4d6-a1668d4aa00f","Type":"ContainerDied","Data":"c91a4181270fe6e3951d362fe783b050b9e5fc5b4be758fbd942ba2af50dfdde"}
Jul 02 21:41:38.504338 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.504322    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:38.505521 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.505490    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 21:41:38.568291 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.568225    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:41:38.630572 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.630544    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9624fa5-352c-4268-b4d6-a1668d4aa00f-proxy-tls\") pod \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") "
Jul 02 21:41:38.630726 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.630615    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bq4k\" (UniqueName: \"kubernetes.io/projected/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-api-access-5bq4k\") pod \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") "
Jul 02 21:41:38.630726 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.630651    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-rbac-proxy-sar-config\") pod \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\" (UID: \"c9624fa5-352c-4268-b4d6-a1668d4aa00f\") "
Jul 02 21:41:38.631150 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.631113    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c9624fa5-352c-4268-b4d6-a1668d4aa00f" (UID: "c9624fa5-352c-4268-b4d6-a1668d4aa00f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:41:38.632676 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.632646    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9624fa5-352c-4268-b4d6-a1668d4aa00f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c9624fa5-352c-4268-b4d6-a1668d4aa00f" (UID: "c9624fa5-352c-4268-b4d6-a1668d4aa00f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:41:38.632777 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.632672    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-api-access-5bq4k" (OuterVolumeSpecName: "kube-api-access-5bq4k") pod "c9624fa5-352c-4268-b4d6-a1668d4aa00f" (UID: "c9624fa5-352c-4268-b4d6-a1668d4aa00f"). InnerVolumeSpecName "kube-api-access-5bq4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:41:38.731496 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.731449    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9624fa5-352c-4268-b4d6-a1668d4aa00f-proxy-tls\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:41:38.731496 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.731491    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bq4k\" (UniqueName: \"kubernetes.io/projected/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-api-access-5bq4k\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:41:38.731496 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:38.731502    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9624fa5-352c-4268-b4d6-a1668d4aa00f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:41:39.510083 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:39.510052    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"
Jul 02 21:41:39.510625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:39.510094    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl" event={"ID":"c9624fa5-352c-4268-b4d6-a1668d4aa00f","Type":"ContainerDied","Data":"17446a81ebe80a93ba48428386d4fa4232bb4fa29d2f0359a804787814a6bdee"}
Jul 02 21:41:39.510625 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:39.510174    2564 scope.go:117] "RemoveContainer" containerID="6207f71468a2a96c92b70ebc788a4d8a70b5f19ba3335a4a553c8186d8d20db4"
Jul 02 21:41:39.510785 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:39.510754    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 21:41:39.519096 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:39.519077    2564 scope.go:117] "RemoveContainer" containerID="c91a4181270fe6e3951d362fe783b050b9e5fc5b4be758fbd942ba2af50dfdde"
Jul 02 21:41:39.531645 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:39.531609    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"]
Jul 02 21:41:39.533963 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:39.533936    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ee70d-predictor-5797bfd779-l6tvl"]
Jul 02 21:41:39.834980 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:39.834898    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" path="/var/lib/kubelet/pods/c9624fa5-352c-4268-b4d6-a1668d4aa00f/volumes"
Jul 02 21:41:44.514811 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:44.514785    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:41:44.515200 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:44.515171    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 21:41:50.750606 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:50.750570    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:52.108342 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:52.108241    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:41:54.515222 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:41:54.515185    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 21:42:00.856678 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:00.856632    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:42:00.974338 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:00.974306    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:42:01.047053 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.047018    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:42:01.062432 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.062400    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"]
Jul 02 21:42:01.062937 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.062888    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kserve-container"
Jul 02 21:42:01.062937 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.062909    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kserve-container"
Jul 02 21:42:01.062937 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.062933    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kube-rbac-proxy"
Jul 02 21:42:01.062937 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.062942    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kube-rbac-proxy"
Jul 02 21:42:01.063241 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.063032    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kserve-container"
Jul 02 21:42:01.063241 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.063045    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9624fa5-352c-4268-b4d6-a1668d4aa00f" containerName="kube-rbac-proxy"
Jul 02 21:42:01.067869 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.067848    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:01.070394 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.070368    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8bf58-predictor-serving-cert\""
Jul 02 21:42:01.070394 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.070379    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8bf58-kube-rbac-proxy-sar-config\""
Jul 02 21:42:01.079448 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.079423    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"]
Jul 02 21:42:01.086541 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.086137    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"]
Jul 02 21:42:01.087018 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.086943    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kserve-container" containerID="cri-o://115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4" gracePeriod=30
Jul 02 21:42:01.087236 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.087208    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kube-rbac-proxy" containerID="cri-o://a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f" gracePeriod=30
Jul 02 21:42:01.114000 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.113919    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:42:01.223409 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.223374    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:01.223592 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.223436    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-proxy-tls\") pod \"error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:01.223592 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.223491    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtlxr\" (UniqueName: \"kubernetes.io/projected/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-api-access-dtlxr\") pod \"error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:01.324521 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.324487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:01.324710 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.324542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-proxy-tls\") pod \"error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:01.324710 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.324576    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtlxr\" (UniqueName: \"kubernetes.io/projected/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-api-access-dtlxr\") pod \"error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:01.324710 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:42:01.324703    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-serving-cert: secret "error-404-isvc-8bf58-predictor-serving-cert" not found
Jul 02 21:42:01.324890 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:42:01.324764    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-proxy-tls podName:7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7 nodeName:}" failed. No retries permitted until 2026-07-02 21:42:01.824746126 +0000 UTC m=+1390.558379423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-proxy-tls") pod "error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" (UID: "7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7") : secret "error-404-isvc-8bf58-predictor-serving-cert" not found
Jul 02 21:42:01.325332 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.325308    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:01.333379 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.333352    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtlxr\" (UniqueName: \"kubernetes.io/projected/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-api-access-dtlxr\") pod \"error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:01.493444 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.493401    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:42:01.610349 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.610309    2564 generic.go:356] "Generic (PLEG): container finished" podID="c0530481-196f-444c-ae68-287e6d9b4138" containerID="a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f" exitCode=2
Jul 02 21:42:01.610510 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.610378    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" event={"ID":"c0530481-196f-444c-ae68-287e6d9b4138","Type":"ContainerDied","Data":"a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f"}
Jul 02 21:42:01.828786 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.828693    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-proxy-tls\") pod \"error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:01.831260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.831227    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-proxy-tls\") pod \"error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:01.984577 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:01.984533    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:02.125697 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:02.125670    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"]
Jul 02 21:42:02.128540 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:42:02.128508    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3bad8d_ffe0_4351_8e8e_0c1b13c833b7.slice/crio-5f5bf7246c94a39a70624ae895a12d5739147abaa7df7210178068f7acbc30aa WatchSource:0}: Error finding container 5f5bf7246c94a39a70624ae895a12d5739147abaa7df7210178068f7acbc30aa: Status 404 returned error can't find the container with id 5f5bf7246c94a39a70624ae895a12d5739147abaa7df7210178068f7acbc30aa
Jul 02 21:42:02.616383 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:02.616337    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" event={"ID":"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7","Type":"ContainerStarted","Data":"ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac"}
Jul 02 21:42:02.616529 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:02.616421    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" event={"ID":"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7","Type":"ContainerStarted","Data":"7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4"}
Jul 02 21:42:02.616529 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:02.616447    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:02.616529 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:02.616466    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" event={"ID":"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7","Type":"ContainerStarted","Data":"5f5bf7246c94a39a70624ae895a12d5739147abaa7df7210178068f7acbc30aa"}
Jul 02 21:42:02.616529 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:02.616504    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:02.617900 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:02.617877    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 21:42:02.636263 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:02.636211    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" podStartSLOduration=1.6361989829999999 podStartE2EDuration="1.636198983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:42:02.634875575 +0000 UTC m=+1391.368508938" watchObservedRunningTime="2026-07-02 21:42:02.636198983 +0000 UTC m=+1391.369832290"
Jul 02 21:42:03.299091 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:03.299051    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 02 21:42:03.620556 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:03.620457    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 21:42:04.239788 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.239764    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:42:04.350441 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.350357    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkm4l\" (UniqueName: \"kubernetes.io/projected/c0530481-196f-444c-ae68-287e6d9b4138-kube-api-access-wkm4l\") pod \"c0530481-196f-444c-ae68-287e6d9b4138\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") "
Jul 02 21:42:04.350441 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.350409    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0530481-196f-444c-ae68-287e6d9b4138-proxy-tls\") pod \"c0530481-196f-444c-ae68-287e6d9b4138\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") "
Jul 02 21:42:04.350441 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.350431    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0530481-196f-444c-ae68-287e6d9b4138-kube-rbac-proxy-sar-config\") pod \"c0530481-196f-444c-ae68-287e6d9b4138\" (UID: \"c0530481-196f-444c-ae68-287e6d9b4138\") "
Jul 02 21:42:04.350922 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.350833    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0530481-196f-444c-ae68-287e6d9b4138-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c0530481-196f-444c-ae68-287e6d9b4138" (UID: "c0530481-196f-444c-ae68-287e6d9b4138"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:42:04.352461 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.352428    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0530481-196f-444c-ae68-287e6d9b4138-kube-api-access-wkm4l" (OuterVolumeSpecName: "kube-api-access-wkm4l") pod "c0530481-196f-444c-ae68-287e6d9b4138" (UID: "c0530481-196f-444c-ae68-287e6d9b4138"). InnerVolumeSpecName "kube-api-access-wkm4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:42:04.352461 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.352451    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0530481-196f-444c-ae68-287e6d9b4138-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0530481-196f-444c-ae68-287e6d9b4138" (UID: "c0530481-196f-444c-ae68-287e6d9b4138"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:42:04.451618 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.451583    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkm4l\" (UniqueName: \"kubernetes.io/projected/c0530481-196f-444c-ae68-287e6d9b4138-kube-api-access-wkm4l\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:42:04.451618 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.451610    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0530481-196f-444c-ae68-287e6d9b4138-proxy-tls\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:42:04.451618 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.451620    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0530481-196f-444c-ae68-287e6d9b4138-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:42:04.515239 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.515200    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 21:42:04.626155 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.626066    2564 generic.go:356] "Generic (PLEG): container finished" podID="c0530481-196f-444c-ae68-287e6d9b4138" containerID="115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4" exitCode=0
Jul 02 21:42:04.626155 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.626112    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" event={"ID":"c0530481-196f-444c-ae68-287e6d9b4138","Type":"ContainerDied","Data":"115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4"}
Jul 02 21:42:04.626155 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.626118    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"
Jul 02 21:42:04.626361 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.626175    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk" event={"ID":"c0530481-196f-444c-ae68-287e6d9b4138","Type":"ContainerDied","Data":"0a14c20d0bb08e720f903037583daf6236ca1850b80df19118e89627342fd568"}
Jul 02 21:42:04.626361 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.626204    2564 scope.go:117] "RemoveContainer" containerID="a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f"
Jul 02 21:42:04.635119 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.635101    2564 scope.go:117] "RemoveContainer" containerID="115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4"
Jul 02 21:42:04.642085 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.642068    2564 scope.go:117] "RemoveContainer" containerID="a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f"
Jul 02 21:42:04.642341 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:42:04.642321    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f\": container with ID starting with a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f not found: ID does not exist" containerID="a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f"
Jul 02 21:42:04.642401 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.642350    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f"} err="failed to get container status \"a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f\": rpc error: code = NotFound desc = could not find container \"a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f\": container with ID starting with a0e53b3ed378fb4ee282209004de476b079d3256b58e54bb6ca197a444d90a9f not found: ID does not exist"
Jul 02 21:42:04.642401 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.642367    2564 scope.go:117] "RemoveContainer" containerID="115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4"
Jul 02 21:42:04.642597 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:42:04.642579    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4\": container with ID starting with 115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4 not found: ID does not exist" containerID="115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4"
Jul 02 21:42:04.642637 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.642602    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4"} err="failed to get container status \"115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4\": rpc error: code = NotFound desc = could not find container \"115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4\": container with ID starting with 115fa79658fc875546d2baf5b6989a63e6c38b0cd1eb6c03bc39b742a62636d4 not found: ID does not exist"
Jul 02 21:42:04.650371 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.650350    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"]
Jul 02 21:42:04.657704 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:04.657683    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1702f-predictor-786664c4bf-btbvk"]
Jul 02 21:42:05.834371 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:05.834340    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0530481-196f-444c-ae68-287e6d9b4138" path="/var/lib/kubelet/pods/c0530481-196f-444c-ae68-287e6d9b4138/volumes"
Jul 02 21:42:08.625084 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:08.625056    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:08.625517 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:08.625493    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 21:42:14.516033 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:14.515968    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 21:42:18.625491 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:18.625450    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 21:42:24.516163 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:24.516131    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:42:28.626446 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:28.626403    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 21:42:35.179790 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:35.179752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:42:35.318437 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:35.318399    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:42:38.625705 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:38.625663    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 21:42:48.626748 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:48.626657    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:42:55.178259 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:42:55.178223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:43:01.047457 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:43:01.047424    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:43:01.156824 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:43:01.156788    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:49:22.369553 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:49:22.369518    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:49:26.480347 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:49:26.480305    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:49:26.581672 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:49:26.581639    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:49:54.156737 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:49:54.156696    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:50:49.848459 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:49.848406    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:50:49.980305 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:49.980272    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:50:50.073182 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.073143    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:50:50.091796 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.091764    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"]
Jul 02 21:50:50.092199 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.092182    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kube-rbac-proxy"
Jul 02 21:50:50.092303 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.092201    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kube-rbac-proxy"
Jul 02 21:50:50.092303 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.092231    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kserve-container"
Jul 02 21:50:50.092303 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.092240    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kserve-container"
Jul 02 21:50:50.092464 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.092313    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kserve-container"
Jul 02 21:50:50.092464 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.092325    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0530481-196f-444c-ae68-287e6d9b4138" containerName="kube-rbac-proxy"
Jul 02 21:50:50.095440 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.095420    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:50.098312 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.098290    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c9d16-kube-rbac-proxy-sar-config\""
Jul 02 21:50:50.098417 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.098297    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c9d16-predictor-serving-cert\""
Jul 02 21:50:50.108107 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.108080    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"]
Jul 02 21:50:50.133342 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.133312    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"]
Jul 02 21:50:50.133570 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.133546    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kserve-container" containerID="cri-o://a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e" gracePeriod=30
Jul 02 21:50:50.133679 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.133560    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kube-rbac-proxy" containerID="cri-o://c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636" gracePeriod=30
Jul 02 21:50:50.147081 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.146960    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:50:50.188340 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.188288    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bb5l\" (UniqueName: \"kubernetes.io/projected/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-api-access-7bb5l\") pod \"error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") " pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:50.188493 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.188408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") " pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:50.188493 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.188445    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/741d6758-2db9-420b-aaae-08ad6ff4c4e6-proxy-tls\") pod \"error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") " pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:50.245644 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.245593    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:50:50.289858 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.289817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") " pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:50.290078 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.289870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/741d6758-2db9-420b-aaae-08ad6ff4c4e6-proxy-tls\") pod \"error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") " pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:50.290078 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:50:50.289987    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-serving-cert: secret "error-404-isvc-c9d16-predictor-serving-cert" not found
Jul 02 21:50:50.290078 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.290033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bb5l\" (UniqueName: \"kubernetes.io/projected/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-api-access-7bb5l\") pod \"error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") " pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:50.290259 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:50:50.290091    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/741d6758-2db9-420b-aaae-08ad6ff4c4e6-proxy-tls podName:741d6758-2db9-420b-aaae-08ad6ff4c4e6 nodeName:}" failed. No retries permitted until 2026-07-02 21:50:50.790059108 +0000 UTC m=+1919.523692393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/741d6758-2db9-420b-aaae-08ad6ff4c4e6-proxy-tls") pod "error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" (UID: "741d6758-2db9-420b-aaae-08ad6ff4c4e6") : secret "error-404-isvc-c9d16-predictor-serving-cert" not found
Jul 02 21:50:50.290649 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.290627    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") " pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:50.298980 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.298954    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bb5l\" (UniqueName: \"kubernetes.io/projected/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-api-access-7bb5l\") pod \"error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") " pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:50.544557 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.544522    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:50:50.728324 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.728290    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:50:50.794887 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.794796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/741d6758-2db9-420b-aaae-08ad6ff4c4e6-proxy-tls\") pod \"error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") " pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:50.797324 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.797291    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/741d6758-2db9-420b-aaae-08ad6ff4c4e6-proxy-tls\") pod \"error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") " pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:50.885207 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.885169    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb66a45d-e080-4406-9218-0c4bf5107548" containerID="c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636" exitCode=2
Jul 02 21:50:50.885568 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:50.885237    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" event={"ID":"cb66a45d-e080-4406-9218-0c4bf5107548","Type":"ContainerDied","Data":"c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636"}
Jul 02 21:50:51.007878 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:51.007840    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:51.139632 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:51.139410    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"]
Jul 02 21:50:51.142108 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:50:51.142080    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741d6758_2db9_420b_aaae_08ad6ff4c4e6.slice/crio-8a732f229610da3ca15f6d50aa4202ec31dd3e1d3b3e0a0eff37c08644df9a44 WatchSource:0}: Error finding container 8a732f229610da3ca15f6d50aa4202ec31dd3e1d3b3e0a0eff37c08644df9a44: Status 404 returned error can't find the container with id 8a732f229610da3ca15f6d50aa4202ec31dd3e1d3b3e0a0eff37c08644df9a44
Jul 02 21:50:51.891687 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:51.891651    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" event={"ID":"741d6758-2db9-420b-aaae-08ad6ff4c4e6","Type":"ContainerStarted","Data":"c009b41297b5027e2ebdf23f429de9ce3a696edc4ac54110fa5e4e63ed37b72e"}
Jul 02 21:50:51.892196 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:51.891722    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" event={"ID":"741d6758-2db9-420b-aaae-08ad6ff4c4e6","Type":"ContainerStarted","Data":"875bebf66225705d4f4e8930e68e8bfda041ea01e1ad88bcf11e8c3245584c57"}
Jul 02 21:50:51.892196 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:51.891753    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" event={"ID":"741d6758-2db9-420b-aaae-08ad6ff4c4e6","Type":"ContainerStarted","Data":"8a732f229610da3ca15f6d50aa4202ec31dd3e1d3b3e0a0eff37c08644df9a44"}
Jul 02 21:50:51.892196 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:51.891786    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:51.892196 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:51.891818    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:51.893554 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:51.893520    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 21:50:51.909239 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:51.909183    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" podStartSLOduration=1.909165115 podStartE2EDuration="1.909165115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:50:51.90890456 +0000 UTC m=+1920.642537910" watchObservedRunningTime="2026-07-02 21:50:51.909165115 +0000 UTC m=+1920.642798423"
Jul 02 21:50:52.896554 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:52.896517    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 21:50:53.295694 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.295661    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:50:53.418050 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.417936    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbr4t\" (UniqueName: \"kubernetes.io/projected/cb66a45d-e080-4406-9218-0c4bf5107548-kube-api-access-rbr4t\") pod \"cb66a45d-e080-4406-9218-0c4bf5107548\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") "
Jul 02 21:50:53.418050 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.417984    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb66a45d-e080-4406-9218-0c4bf5107548-kube-rbac-proxy-sar-config\") pod \"cb66a45d-e080-4406-9218-0c4bf5107548\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") "
Jul 02 21:50:53.418289 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.418066    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb66a45d-e080-4406-9218-0c4bf5107548-proxy-tls\") pod \"cb66a45d-e080-4406-9218-0c4bf5107548\" (UID: \"cb66a45d-e080-4406-9218-0c4bf5107548\") "
Jul 02 21:50:53.418408 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.418380    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb66a45d-e080-4406-9218-0c4bf5107548-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb66a45d-e080-4406-9218-0c4bf5107548" (UID: "cb66a45d-e080-4406-9218-0c4bf5107548"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:50:53.420069 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.420046    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb66a45d-e080-4406-9218-0c4bf5107548-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb66a45d-e080-4406-9218-0c4bf5107548" (UID: "cb66a45d-e080-4406-9218-0c4bf5107548"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:50:53.420248 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.420220    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb66a45d-e080-4406-9218-0c4bf5107548-kube-api-access-rbr4t" (OuterVolumeSpecName: "kube-api-access-rbr4t") pod "cb66a45d-e080-4406-9218-0c4bf5107548" (UID: "cb66a45d-e080-4406-9218-0c4bf5107548"). InnerVolumeSpecName "kube-api-access-rbr4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:50:53.519506 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.519468    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbr4t\" (UniqueName: \"kubernetes.io/projected/cb66a45d-e080-4406-9218-0c4bf5107548-kube-api-access-rbr4t\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:50:53.519506 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.519500    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb66a45d-e080-4406-9218-0c4bf5107548-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:50:53.519506 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.519510    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb66a45d-e080-4406-9218-0c4bf5107548-proxy-tls\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:50:53.902365 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.902329    2564 generic.go:356] "Generic (PLEG): container finished" podID="cb66a45d-e080-4406-9218-0c4bf5107548" containerID="a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e" exitCode=0
Jul 02 21:50:53.902755 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.902385    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"
Jul 02 21:50:53.902755 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.902418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" event={"ID":"cb66a45d-e080-4406-9218-0c4bf5107548","Type":"ContainerDied","Data":"a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e"}
Jul 02 21:50:53.902755 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.902477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr" event={"ID":"cb66a45d-e080-4406-9218-0c4bf5107548","Type":"ContainerDied","Data":"96f2f4a4cff7c6038bf304c691cd3c74b6e7012164d54d135d8df19a032e51e9"}
Jul 02 21:50:53.902755 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.902503    2564 scope.go:117] "RemoveContainer" containerID="c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636"
Jul 02 21:50:53.912165 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.912133    2564 scope.go:117] "RemoveContainer" containerID="a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e"
Jul 02 21:50:53.919410 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.919391    2564 scope.go:117] "RemoveContainer" containerID="c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636"
Jul 02 21:50:53.919676 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:50:53.919657    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636\": container with ID starting with c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636 not found: ID does not exist" containerID="c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636"
Jul 02 21:50:53.919739 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.919684    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636"} err="failed to get container status \"c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636\": rpc error: code = NotFound desc = could not find container \"c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636\": container with ID starting with c9ecf6896e172d7f78ceea79da833107b345817645b96b8c2cb352e2f2e22636 not found: ID does not exist"
Jul 02 21:50:53.919739 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.919700    2564 scope.go:117] "RemoveContainer" containerID="a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e"
Jul 02 21:50:53.919938 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:50:53.919923    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e\": container with ID starting with a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e not found: ID does not exist" containerID="a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e"
Jul 02 21:50:53.919982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.919942    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e"} err="failed to get container status \"a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e\": rpc error: code = NotFound desc = could not find container \"a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e\": container with ID starting with a90762dab5964620c9bb82b56f5382e48a63164aecf6ec718c9af152631b0e5e not found: ID does not exist"
Jul 02 21:50:53.922959 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.922938    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"]
Jul 02 21:50:53.926785 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:53.926766    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87dcc-predictor-5b4bfbcdc9-c5zlr"]
Jul 02 21:50:55.834780 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:55.834749    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" path="/var/lib/kubelet/pods/cb66a45d-e080-4406-9218-0c4bf5107548/volumes"
Jul 02 21:50:57.900363 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:57.900336    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:50:57.900868 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:50:57.900842    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 21:51:07.901335 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:07.901292    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 21:51:15.765432 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:15.765397    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:51:15.884126 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:15.884076    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:51:15.988566 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:15.988521    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:51:16.001167 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.001122    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"]
Jul 02 21:51:16.001471 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.001456    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kserve-container"
Jul 02 21:51:16.001471 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.001472    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kserve-container"
Jul 02 21:51:16.001594 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.001488    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kube-rbac-proxy"
Jul 02 21:51:16.001594 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.001494    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kube-rbac-proxy"
Jul 02 21:51:16.001594 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.001549    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kserve-container"
Jul 02 21:51:16.001594 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.001558    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb66a45d-e080-4406-9218-0c4bf5107548" containerName="kube-rbac-proxy"
Jul 02 21:51:16.005792 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.005772    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:16.008414 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.008393    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-76a5f-predictor-serving-cert\""
Jul 02 21:51:16.008543 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.008411    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-76a5f-kube-rbac-proxy-sar-config\""
Jul 02 21:51:16.015490 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.015233    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80691226-a702-4432-b4c1-08223db2b3db-proxy-tls\") pod \"error-404-isvc-76a5f-predictor-578849c975-9gcw7\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") " pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:16.015490 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.015348    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w4p9\" (UniqueName: \"kubernetes.io/projected/80691226-a702-4432-b4c1-08223db2b3db-kube-api-access-2w4p9\") pod \"error-404-isvc-76a5f-predictor-578849c975-9gcw7\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") " pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:16.015490 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.015397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80691226-a702-4432-b4c1-08223db2b3db-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-76a5f-predictor-578849c975-9gcw7\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") " pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:16.019552 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.016318    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"]
Jul 02 21:51:16.045573 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.045540    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"]
Jul 02 21:51:16.046072 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.046019    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kserve-container" containerID="cri-o://7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4" gracePeriod=30
Jul 02 21:51:16.046344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.046154    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kube-rbac-proxy" containerID="cri-o://ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac" gracePeriod=30
Jul 02 21:51:16.081865 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.081836    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:51:16.116140 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.116105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80691226-a702-4432-b4c1-08223db2b3db-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-76a5f-predictor-578849c975-9gcw7\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") " pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:16.116281 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.116169    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80691226-a702-4432-b4c1-08223db2b3db-proxy-tls\") pod \"error-404-isvc-76a5f-predictor-578849c975-9gcw7\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") " pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:16.116281 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.116246    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w4p9\" (UniqueName: \"kubernetes.io/projected/80691226-a702-4432-b4c1-08223db2b3db-kube-api-access-2w4p9\") pod \"error-404-isvc-76a5f-predictor-578849c975-9gcw7\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") " pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:16.116399 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:51:16.116333    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-serving-cert: secret "error-404-isvc-76a5f-predictor-serving-cert" not found
Jul 02 21:51:16.116452 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:51:16.116409    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80691226-a702-4432-b4c1-08223db2b3db-proxy-tls podName:80691226-a702-4432-b4c1-08223db2b3db nodeName:}" failed. No retries permitted until 2026-07-02 21:51:16.616387761 +0000 UTC m=+1945.350021055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/80691226-a702-4432-b4c1-08223db2b3db-proxy-tls") pod "error-404-isvc-76a5f-predictor-578849c975-9gcw7" (UID: "80691226-a702-4432-b4c1-08223db2b3db") : secret "error-404-isvc-76a5f-predictor-serving-cert" not found
Jul 02 21:51:16.116782 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.116757    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80691226-a702-4432-b4c1-08223db2b3db-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-76a5f-predictor-578849c975-9gcw7\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") " pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:16.126212 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.126185    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w4p9\" (UniqueName: \"kubernetes.io/projected/80691226-a702-4432-b4c1-08223db2b3db-kube-api-access-2w4p9\") pod \"error-404-isvc-76a5f-predictor-578849c975-9gcw7\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") " pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:16.144355 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.144328    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:51:16.345099 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.345011    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:51:16.445473 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.445432    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:51:16.620660 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.620560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80691226-a702-4432-b4c1-08223db2b3db-proxy-tls\") pod \"error-404-isvc-76a5f-predictor-578849c975-9gcw7\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") " pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:16.622934 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.622902    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80691226-a702-4432-b4c1-08223db2b3db-proxy-tls\") pod \"error-404-isvc-76a5f-predictor-578849c975-9gcw7\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") " pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:16.918698 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:16.918605    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:17.011111 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:17.011071    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerID="ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac" exitCode=2
Jul 02 21:51:17.011295 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:17.011118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" event={"ID":"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7","Type":"ContainerDied","Data":"ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac"}
Jul 02 21:51:17.057689 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:17.057660    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"]
Jul 02 21:51:17.059292 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:51:17.059264    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80691226_a702_4432_b4c1_08223db2b3db.slice/crio-a5165756ca1faa57d05a93b88f4995f86ed834c2562e7c9062648325ff1dcdd8 WatchSource:0}: Error finding container a5165756ca1faa57d05a93b88f4995f86ed834c2562e7c9062648325ff1dcdd8: Status 404 returned error can't find the container with id a5165756ca1faa57d05a93b88f4995f86ed834c2562e7c9062648325ff1dcdd8
Jul 02 21:51:17.901385 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:17.901346    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 21:51:18.017665 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:18.017627    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" event={"ID":"80691226-a702-4432-b4c1-08223db2b3db","Type":"ContainerStarted","Data":"54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3"}
Jul 02 21:51:18.018084 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:18.017683    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" event={"ID":"80691226-a702-4432-b4c1-08223db2b3db","Type":"ContainerStarted","Data":"8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c"}
Jul 02 21:51:18.018084 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:18.017702    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" event={"ID":"80691226-a702-4432-b4c1-08223db2b3db","Type":"ContainerStarted","Data":"a5165756ca1faa57d05a93b88f4995f86ed834c2562e7c9062648325ff1dcdd8"}
Jul 02 21:51:18.018084 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:18.017726    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:18.018084 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:18.017747    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:18.019344 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:18.019316    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 21:51:18.037875 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:18.037822    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" podStartSLOduration=3.037807761 podStartE2EDuration="3.037807761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:51:18.035888905 +0000 UTC m=+1946.769522236" watchObservedRunningTime="2026-07-02 21:51:18.037807761 +0000 UTC m=+1946.771441118"
Jul 02 21:51:18.621463 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:18.621426    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 02 21:51:18.625982 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:18.625957    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 21:51:19.022201 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:19.022162    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 21:51:19.410688 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:19.410661    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:51:19.444046 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:19.444016    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-rbac-proxy-sar-config\") pod \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") "
Jul 02 21:51:19.444173 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:19.444086    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-proxy-tls\") pod \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") "
Jul 02 21:51:19.444173 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:19.444134    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtlxr\" (UniqueName: \"kubernetes.io/projected/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-api-access-dtlxr\") pod \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\" (UID: \"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7\") "
Jul 02 21:51:19.444403 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:19.444373    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" (UID: "7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:51:19.446190 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:19.446160    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" (UID: "7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:51:19.446190 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:19.446173    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-api-access-dtlxr" (OuterVolumeSpecName: "kube-api-access-dtlxr") pod "7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" (UID: "7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7"). InnerVolumeSpecName "kube-api-access-dtlxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:51:19.544632 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:19.544596    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-proxy-tls\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:51:19.544632 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:19.544629    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtlxr\" (UniqueName: \"kubernetes.io/projected/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-api-access-dtlxr\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:51:19.544822 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:19.544642    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:51:20.027916 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.027883    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerID="7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4" exitCode=0
Jul 02 21:51:20.028351 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.027933    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"
Jul 02 21:51:20.028351 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.027958    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" event={"ID":"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7","Type":"ContainerDied","Data":"7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4"}
Jul 02 21:51:20.028351 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.028045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm" event={"ID":"7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7","Type":"ContainerDied","Data":"5f5bf7246c94a39a70624ae895a12d5739147abaa7df7210178068f7acbc30aa"}
Jul 02 21:51:20.028351 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.028085    2564 scope.go:117] "RemoveContainer" containerID="ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac"
Jul 02 21:51:20.035786 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.035767    2564 scope.go:117] "RemoveContainer" containerID="7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4"
Jul 02 21:51:20.043061 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.043039    2564 scope.go:117] "RemoveContainer" containerID="ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac"
Jul 02 21:51:20.043345 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:51:20.043322    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac\": container with ID starting with ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac not found: ID does not exist" containerID="ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac"
Jul 02 21:51:20.043427 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.043353    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac"} err="failed to get container status \"ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac\": rpc error: code = NotFound desc = could not find container \"ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac\": container with ID starting with ce8544cb95a910719e8e875d8e46d6e4d40f202d7806aded3fd32221b34b6fac not found: ID does not exist"
Jul 02 21:51:20.043427 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.043375    2564 scope.go:117] "RemoveContainer" containerID="7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4"
Jul 02 21:51:20.043648 ip-10-0-130-121 kubenswrapper[2564]: E0702 21:51:20.043628    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4\": container with ID starting with 7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4 not found: ID does not exist" containerID="7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4"
Jul 02 21:51:20.043693 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.043655    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4"} err="failed to get container status \"7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4\": rpc error: code = NotFound desc = could not find container \"7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4\": container with ID starting with 7d9671e208dd3adda204e1c6504c9fd3fab6e3bbb56e9c9d328064b21bb929f4 not found: ID does not exist"
Jul 02 21:51:20.046403 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.046381    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"]
Jul 02 21:51:20.050730 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:20.050711    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8bf58-predictor-5d6f9ddb7f-l7cxm"]
Jul 02 21:51:21.839198 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:21.839165    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" path="/var/lib/kubelet/pods/7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7/volumes"
Jul 02 21:51:24.026616 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:24.026581    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:51:24.027171 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:24.027142    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 21:51:27.900776 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:27.900732    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 21:51:34.027205 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:34.027166    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 21:51:37.902221 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:37.902190    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:51:44.027863 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:44.027775    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 21:51:50.047528 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:50.047490    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:51:50.391637 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:50.391544    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:51:54.027355 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:51:54.027318    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 21:52:00.141756 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.141643    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:52:00.205134 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.205096    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"]
Jul 02 21:52:00.205405 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.205377    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kserve-container" containerID="cri-o://875bebf66225705d4f4e8930e68e8bfda041ea01e1ad88bcf11e8c3245584c57" gracePeriod=30
Jul 02 21:52:00.205807 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.205648    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kube-rbac-proxy" containerID="cri-o://c009b41297b5027e2ebdf23f429de9ce3a696edc4ac54110fa5e4e63ed37b72e" gracePeriod=30
Jul 02 21:52:00.236028 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.235970    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:52:00.309314 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.309281    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:52:00.369883 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.369846    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"]
Jul 02 21:52:00.370319 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.370299    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kube-rbac-proxy"
Jul 02 21:52:00.370319 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.370318    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kube-rbac-proxy"
Jul 02 21:52:00.370319 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.370327    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kserve-container"
Jul 02 21:52:00.370537 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.370333    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kserve-container"
Jul 02 21:52:00.370537 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.370401    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kserve-container"
Jul 02 21:52:00.370537 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.370412    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b3bad8d-ffe0-4351-8e8e-0c1b13c833b7" containerName="kube-rbac-proxy"
Jul 02 21:52:00.373455 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.373439    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:00.376185 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.376163    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-67ac0-predictor-serving-cert\""
Jul 02 21:52:00.376274 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.376183    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-67ac0-kube-rbac-proxy-sar-config\""
Jul 02 21:52:00.383258 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.383236    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"]
Jul 02 21:52:00.390875 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.390843    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:52:00.494448 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.494413    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42q9b\" (UniqueName: \"kubernetes.io/projected/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-api-access-42q9b\") pod \"error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:00.494627 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.494474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af0d93e-02b1-4c46-832a-6580f162dfbb-proxy-tls\") pod \"error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:00.494627 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.494530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:00.580525 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.580493    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:52:00.595785 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.595750    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42q9b\" (UniqueName: \"kubernetes.io/projected/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-api-access-42q9b\") pod \"error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:00.595957 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.595817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af0d93e-02b1-4c46-832a-6580f162dfbb-proxy-tls\") pod \"error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:00.595957 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.595854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:00.596460 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.596438    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:00.598260 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.598243    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af0d93e-02b1-4c46-832a-6580f162dfbb-proxy-tls\") pod \"error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:00.605173 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.605144    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42q9b\" (UniqueName: \"kubernetes.io/projected/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-api-access-42q9b\") pod \"error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") " pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:00.684590 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.684552    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:00.817605 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:00.817580    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"]
Jul 02 21:52:00.819919 ip-10-0-130-121 kubenswrapper[2564]: W0702 21:52:00.819886    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af0d93e_02b1_4c46_832a_6580f162dfbb.slice/crio-7fcd624e4413e11e962468cac27cc6c602cd02b242070c95f98ba52d80a57f22 WatchSource:0}: Error finding container 7fcd624e4413e11e962468cac27cc6c602cd02b242070c95f98ba52d80a57f22: Status 404 returned error can't find the container with id 7fcd624e4413e11e962468cac27cc6c602cd02b242070c95f98ba52d80a57f22
Jul 02 21:52:01.213550 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:01.213517    2564 generic.go:356] "Generic (PLEG): container finished" podID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerID="c009b41297b5027e2ebdf23f429de9ce3a696edc4ac54110fa5e4e63ed37b72e" exitCode=2
Jul 02 21:52:01.213924 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:01.213593    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" event={"ID":"741d6758-2db9-420b-aaae-08ad6ff4c4e6","Type":"ContainerDied","Data":"c009b41297b5027e2ebdf23f429de9ce3a696edc4ac54110fa5e4e63ed37b72e"}
Jul 02 21:52:01.215361 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:01.215341    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" event={"ID":"4af0d93e-02b1-4c46-832a-6580f162dfbb","Type":"ContainerStarted","Data":"e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09"}
Jul 02 21:52:01.215468 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:01.215388    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" event={"ID":"4af0d93e-02b1-4c46-832a-6580f162dfbb","Type":"ContainerStarted","Data":"6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825"}
Jul 02 21:52:01.215468 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:01.215412    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" event={"ID":"4af0d93e-02b1-4c46-832a-6580f162dfbb","Type":"ContainerStarted","Data":"7fcd624e4413e11e962468cac27cc6c602cd02b242070c95f98ba52d80a57f22"}
Jul 02 21:52:01.215468 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:01.215459    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:01.233971 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:01.233922    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" podStartSLOduration=1.233909445 podStartE2EDuration="1.233909445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 21:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 21:52:01.232119737 +0000 UTC m=+1989.965753071" watchObservedRunningTime="2026-07-02 21:52:01.233909445 +0000 UTC m=+1989.967542752"
Jul 02 21:52:02.220056 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:02.220022    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:02.221518 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:02.221486    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 21:52:02.897180 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:02.897135    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 02 21:52:03.225527 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.225494    2564 generic.go:356] "Generic (PLEG): container finished" podID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerID="875bebf66225705d4f4e8930e68e8bfda041ea01e1ad88bcf11e8c3245584c57" exitCode=0
Jul 02 21:52:03.225903 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.225573    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" event={"ID":"741d6758-2db9-420b-aaae-08ad6ff4c4e6","Type":"ContainerDied","Data":"875bebf66225705d4f4e8930e68e8bfda041ea01e1ad88bcf11e8c3245584c57"}
Jul 02 21:52:03.226072 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.226048    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 21:52:03.355441 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.355414    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:52:03.520816 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.520779    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bb5l\" (UniqueName: \"kubernetes.io/projected/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-api-access-7bb5l\") pod \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") "
Jul 02 21:52:03.521018 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.520848    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-rbac-proxy-sar-config\") pod \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") "
Jul 02 21:52:03.521018 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.520885    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/741d6758-2db9-420b-aaae-08ad6ff4c4e6-proxy-tls\") pod \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\" (UID: \"741d6758-2db9-420b-aaae-08ad6ff4c4e6\") "
Jul 02 21:52:03.521287 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.521260    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "741d6758-2db9-420b-aaae-08ad6ff4c4e6" (UID: "741d6758-2db9-420b-aaae-08ad6ff4c4e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 21:52:03.522894 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.522871    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741d6758-2db9-420b-aaae-08ad6ff4c4e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "741d6758-2db9-420b-aaae-08ad6ff4c4e6" (UID: "741d6758-2db9-420b-aaae-08ad6ff4c4e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 21:52:03.522953 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.522873    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-api-access-7bb5l" (OuterVolumeSpecName: "kube-api-access-7bb5l") pod "741d6758-2db9-420b-aaae-08ad6ff4c4e6" (UID: "741d6758-2db9-420b-aaae-08ad6ff4c4e6"). InnerVolumeSpecName "kube-api-access-7bb5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 21:52:03.622482 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.622447    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bb5l\" (UniqueName: \"kubernetes.io/projected/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-api-access-7bb5l\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:52:03.622482 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.622476    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/741d6758-2db9-420b-aaae-08ad6ff4c4e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:52:03.622482 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:03.622488    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/741d6758-2db9-420b-aaae-08ad6ff4c4e6-proxy-tls\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 21:52:04.028161 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:04.028132    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 21:52:04.230957 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:04.230926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl" event={"ID":"741d6758-2db9-420b-aaae-08ad6ff4c4e6","Type":"ContainerDied","Data":"8a732f229610da3ca15f6d50aa4202ec31dd3e1d3b3e0a0eff37c08644df9a44"}
Jul 02 21:52:04.230957 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:04.230955    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"
Jul 02 21:52:04.231447 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:04.231028    2564 scope.go:117] "RemoveContainer" containerID="c009b41297b5027e2ebdf23f429de9ce3a696edc4ac54110fa5e4e63ed37b72e"
Jul 02 21:52:04.239166 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:04.239147    2564 scope.go:117] "RemoveContainer" containerID="875bebf66225705d4f4e8930e68e8bfda041ea01e1ad88bcf11e8c3245584c57"
Jul 02 21:52:04.248648 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:04.248625    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"]
Jul 02 21:52:04.252522 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:04.252499    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c9d16-predictor-d5f45cf87-t2ssl"]
Jul 02 21:52:05.834565 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:05.834532    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" path="/var/lib/kubelet/pods/741d6758-2db9-420b-aaae-08ad6ff4c4e6/volumes"
Jul 02 21:52:08.230949 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:08.230917    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:52:08.231437 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:08.231406    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 21:52:15.983070 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:15.983035    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:52:16.084533 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:16.084498    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:52:18.232186 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:18.232142    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 21:52:28.232269 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:28.232226    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 21:52:38.232231 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:38.232187    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 21:52:48.232846 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:52:48.232814    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 21:53:00.340131 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:53:00.340076    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:53:00.438419 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:53:00.438384    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:54:48.237136 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:54:48.237066    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:57:16.280787 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:57:16.280740    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:57:39.210492 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:57:39.210456    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 21:58:20.029514 ip-10-0-130-121 kubenswrapper[2564]: I0702 21:58:20.029479    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:01:15.002206 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:15.002102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:01:15.126144 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:15.126104    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"]
Jul 02 22:01:15.126448 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:15.126420    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kserve-container" containerID="cri-o://6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825" gracePeriod=30
Jul 02 22:01:15.126548 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:15.126457    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kube-rbac-proxy" containerID="cri-o://e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09" gracePeriod=30
Jul 02 22:01:15.574922 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:15.574883    2564 generic.go:356] "Generic (PLEG): container finished" podID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerID="e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09" exitCode=2
Jul 02 22:01:15.575099 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:15.574932    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" event={"ID":"4af0d93e-02b1-4c46-832a-6580f162dfbb","Type":"ContainerDied","Data":"e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09"}
Jul 02 22:01:16.504681 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:16.504645    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:01:18.280548 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.280521    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 22:01:18.439697 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.439615    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42q9b\" (UniqueName: \"kubernetes.io/projected/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-api-access-42q9b\") pod \"4af0d93e-02b1-4c46-832a-6580f162dfbb\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") "
Jul 02 22:01:18.439697 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.439681    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af0d93e-02b1-4c46-832a-6580f162dfbb-proxy-tls\") pod \"4af0d93e-02b1-4c46-832a-6580f162dfbb\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") "
Jul 02 22:01:18.439930 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.439714    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-rbac-proxy-sar-config\") pod \"4af0d93e-02b1-4c46-832a-6580f162dfbb\" (UID: \"4af0d93e-02b1-4c46-832a-6580f162dfbb\") "
Jul 02 22:01:18.440165 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.440132    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4af0d93e-02b1-4c46-832a-6580f162dfbb" (UID: "4af0d93e-02b1-4c46-832a-6580f162dfbb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 22:01:18.441779 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.441748    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-api-access-42q9b" (OuterVolumeSpecName: "kube-api-access-42q9b") pod "4af0d93e-02b1-4c46-832a-6580f162dfbb" (UID: "4af0d93e-02b1-4c46-832a-6580f162dfbb"). InnerVolumeSpecName "kube-api-access-42q9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 22:01:18.441868 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.441798    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af0d93e-02b1-4c46-832a-6580f162dfbb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4af0d93e-02b1-4c46-832a-6580f162dfbb" (UID: "4af0d93e-02b1-4c46-832a-6580f162dfbb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 22:01:18.540793 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.540761    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af0d93e-02b1-4c46-832a-6580f162dfbb-proxy-tls\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 22:01:18.540793 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.540789    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 22:01:18.540793 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.540799    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42q9b\" (UniqueName: \"kubernetes.io/projected/4af0d93e-02b1-4c46-832a-6580f162dfbb-kube-api-access-42q9b\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 22:01:18.589363 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.589335    2564 generic.go:356] "Generic (PLEG): container finished" podID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerID="6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825" exitCode=0
Jul 02 22:01:18.589488 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.589381    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"
Jul 02 22:01:18.589488 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.589412    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" event={"ID":"4af0d93e-02b1-4c46-832a-6580f162dfbb","Type":"ContainerDied","Data":"6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825"}
Jul 02 22:01:18.589488 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.589459    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" event={"ID":"4af0d93e-02b1-4c46-832a-6580f162dfbb","Type":"ContainerDied","Data":"7fcd624e4413e11e962468cac27cc6c602cd02b242070c95f98ba52d80a57f22"}
Jul 02 22:01:18.589488 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.589482    2564 scope.go:117] "RemoveContainer" containerID="e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09"
Jul 02 22:01:18.597762 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.597744    2564 scope.go:117] "RemoveContainer" containerID="6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825"
Jul 02 22:01:18.604733 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.604715    2564 scope.go:117] "RemoveContainer" containerID="e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09"
Jul 02 22:01:18.604988 ip-10-0-130-121 kubenswrapper[2564]: E0702 22:01:18.604958    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09\": container with ID starting with e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09 not found: ID does not exist" containerID="e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09"
Jul 02 22:01:18.605055 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.605015    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09"} err="failed to get container status \"e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09\": rpc error: code = NotFound desc = could not find container \"e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09\": container with ID starting with e8d5da0298c9b4897628411b5263b9c2b0e05817ee9ba688cd46267b92f29d09 not found: ID does not exist"
Jul 02 22:01:18.605055 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.605032    2564 scope.go:117] "RemoveContainer" containerID="6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825"
Jul 02 22:01:18.605290 ip-10-0-130-121 kubenswrapper[2564]: E0702 22:01:18.605272    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825\": container with ID starting with 6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825 not found: ID does not exist" containerID="6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825"
Jul 02 22:01:18.605329 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.605296    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825"} err="failed to get container status \"6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825\": rpc error: code = NotFound desc = could not find container \"6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825\": container with ID starting with 6eb0734213c2ec3115de929a17327878394ea97ade845f0bb6124e68040a0825 not found: ID does not exist"
Jul 02 22:01:18.609888 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.609865    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"]
Jul 02 22:01:18.613825 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:18.613802    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq"]
Jul 02 22:01:19.226940 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:19.226897    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 02 22:01:19.232096 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:19.232067    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67ac0-predictor-6479ffbcb7-wlhcq" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: i/o timeout"
Jul 02 22:01:19.834444 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:01:19.834412    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" path="/var/lib/kubelet/pods/4af0d93e-02b1-4c46-832a-6580f162dfbb/volumes"
Jul 02 22:03:29.525024 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:03:29.524979    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:03:51.584375 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:03:51.584292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:08:35.293340 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:35.293304    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:08:35.430423 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:35.430383    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"]
Jul 02 22:08:35.430687 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:35.430640    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kserve-container" containerID="cri-o://8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c" gracePeriod=30
Jul 02 22:08:35.430780 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:35.430701    2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kube-rbac-proxy" containerID="cri-o://54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3" gracePeriod=30
Jul 02 22:08:36.422259 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:36.422222    2564 generic.go:356] "Generic (PLEG): container finished" podID="80691226-a702-4432-b4c1-08223db2b3db" containerID="54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3" exitCode=2
Jul 02 22:08:36.422624 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:36.422296    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" event={"ID":"80691226-a702-4432-b4c1-08223db2b3db","Type":"ContainerDied","Data":"54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3"}
Jul 02 22:08:36.589540 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:36.589505    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:08:38.579943 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:38.579923    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 22:08:38.585642 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:38.585616    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w4p9\" (UniqueName: \"kubernetes.io/projected/80691226-a702-4432-b4c1-08223db2b3db-kube-api-access-2w4p9\") pod \"80691226-a702-4432-b4c1-08223db2b3db\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") "
Jul 02 22:08:38.585750 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:38.585647    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80691226-a702-4432-b4c1-08223db2b3db-proxy-tls\") pod \"80691226-a702-4432-b4c1-08223db2b3db\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") "
Jul 02 22:08:38.585750 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:38.585693    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80691226-a702-4432-b4c1-08223db2b3db-kube-rbac-proxy-sar-config\") pod \"80691226-a702-4432-b4c1-08223db2b3db\" (UID: \"80691226-a702-4432-b4c1-08223db2b3db\") "
Jul 02 22:08:38.586110 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:38.586089    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80691226-a702-4432-b4c1-08223db2b3db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80691226-a702-4432-b4c1-08223db2b3db" (UID: "80691226-a702-4432-b4c1-08223db2b3db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 22:08:38.587519 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:38.587487    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80691226-a702-4432-b4c1-08223db2b3db-kube-api-access-2w4p9" (OuterVolumeSpecName: "kube-api-access-2w4p9") pod "80691226-a702-4432-b4c1-08223db2b3db" (UID: "80691226-a702-4432-b4c1-08223db2b3db"). InnerVolumeSpecName "kube-api-access-2w4p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 22:08:38.587601 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:38.587540    2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80691226-a702-4432-b4c1-08223db2b3db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80691226-a702-4432-b4c1-08223db2b3db" (UID: "80691226-a702-4432-b4c1-08223db2b3db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 22:08:38.687064 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:38.686968    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80691226-a702-4432-b4c1-08223db2b3db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 22:08:38.687064 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:38.687018    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w4p9\" (UniqueName: \"kubernetes.io/projected/80691226-a702-4432-b4c1-08223db2b3db-kube-api-access-2w4p9\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 22:08:38.687064 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:38.687028    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80691226-a702-4432-b4c1-08223db2b3db-proxy-tls\") on node \"ip-10-0-130-121.ec2.internal\" DevicePath \"\""
Jul 02 22:08:39.437280 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.437246    2564 generic.go:356] "Generic (PLEG): container finished" podID="80691226-a702-4432-b4c1-08223db2b3db" containerID="8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c" exitCode=0
Jul 02 22:08:39.437433 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.437294    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"
Jul 02 22:08:39.437433 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.437317    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" event={"ID":"80691226-a702-4432-b4c1-08223db2b3db","Type":"ContainerDied","Data":"8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c"}
Jul 02 22:08:39.437433 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.437367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7" event={"ID":"80691226-a702-4432-b4c1-08223db2b3db","Type":"ContainerDied","Data":"a5165756ca1faa57d05a93b88f4995f86ed834c2562e7c9062648325ff1dcdd8"}
Jul 02 22:08:39.437433 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.437393    2564 scope.go:117] "RemoveContainer" containerID="54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3"
Jul 02 22:08:39.446428 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.446410    2564 scope.go:117] "RemoveContainer" containerID="8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c"
Jul 02 22:08:39.453674 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.453658    2564 scope.go:117] "RemoveContainer" containerID="54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3"
Jul 02 22:08:39.453906 ip-10-0-130-121 kubenswrapper[2564]: E0702 22:08:39.453886    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3\": container with ID starting with 54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3 not found: ID does not exist" containerID="54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3"
Jul 02 22:08:39.453945 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.453914    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3"} err="failed to get container status \"54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3\": rpc error: code = NotFound desc = could not find container \"54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3\": container with ID starting with 54f0f1d37a434b12b245a3c755ad87faf5fb91c19748bbc3aa103d21dcbe59e3 not found: ID does not exist"
Jul 02 22:08:39.453945 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.453931    2564 scope.go:117] "RemoveContainer" containerID="8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c"
Jul 02 22:08:39.454183 ip-10-0-130-121 kubenswrapper[2564]: E0702 22:08:39.454166    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c\": container with ID starting with 8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c not found: ID does not exist" containerID="8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c"
Jul 02 22:08:39.454232 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.454188    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c"} err="failed to get container status \"8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c\": rpc error: code = NotFound desc = could not find container \"8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c\": container with ID starting with 8267f2cbc65a33846c1d3780a5f0b7c5e4f13e526b47fdf304bf2f611082256c not found: ID does not exist"
Jul 02 22:08:39.459234 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.459213    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"]
Jul 02 22:08:39.464816 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.464795    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a5f-predictor-578849c975-9gcw7"]
Jul 02 22:08:39.834046 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:08:39.834013    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80691226-a702-4432-b4c1-08223db2b3db" path="/var/lib/kubelet/pods/80691226-a702-4432-b4c1-08223db2b3db/volumes"
Jul 02 22:09:06.518291 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:06.518240    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:09:06.692118 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:06.692076    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:09:06.758492 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:06.758454    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:09:09.206895 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:09.206860    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mdqbk_71429679-06c0-480b-8c79-08a2c5682ca1/global-pull-secret-syncer/0.log"
Jul 02 22:09:09.264799 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:09.264774    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log"
Jul 02 22:09:09.354212 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:09.354142    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-121.ec2.internal_fddbb2f027add0a069d4010707bb8cdd/haproxy/0.log"
Jul 02 22:09:12.235510 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:12.235479    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:09:12.303896 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:12.303867    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nm8zf_e7012e89-72a9-4dd4-bc20-10759e43a4dc/konnectivity-agent/0.log" limit=1024
Jul 02 22:09:14.217393 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:14.217359    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84b9bd8c9c-qm4kg_c10ba939-b295-4a3b-a3e0-68a34e5bff76/metrics-server/0.log"
Jul 02 22:09:14.361554 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:14.361525    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpxbf_08e8efc8-577a-42b2-be73-75dad1ba7702/node-exporter/0.log"
Jul 02 22:09:14.385620 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:14.385591    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpxbf_08e8efc8-577a-42b2-be73-75dad1ba7702/kube-rbac-proxy/0.log"
Jul 02 22:09:14.414058 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:14.414022    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpxbf_08e8efc8-577a-42b2-be73-75dad1ba7702/init-textfile/0.log"
Jul 02 22:09:14.914562 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:14.914532    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-rtdsl_a7dd804b-09c4-4d04-b3c4-ec3f1fdaef06/prometheus-operator-admission-webhook/0.log"
Jul 02 22:09:16.463213 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:16.463187    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-shwvz_287c4ada-4462-484f-9832-b617173a665f/networking-console-plugin/0.log"
Jul 02 22:09:17.341901 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.341864    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"]
Jul 02 22:09:17.342206 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342192    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kserve-container"
Jul 02 22:09:17.342264 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342207    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kserve-container"
Jul 02 22:09:17.342264 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342223    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kserve-container"
Jul 02 22:09:17.342264 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342229    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kserve-container"
Jul 02 22:09:17.342264 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342239    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kube-rbac-proxy"
Jul 02 22:09:17.342264 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342245    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kube-rbac-proxy"
Jul 02 22:09:17.342264 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342252    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kube-rbac-proxy"
Jul 02 22:09:17.342264 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342258    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kube-rbac-proxy"
Jul 02 22:09:17.342264 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342264    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kserve-container"
Jul 02 22:09:17.342558 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342270    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kserve-container"
Jul 02 22:09:17.342558 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342278    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kube-rbac-proxy"
Jul 02 22:09:17.342558 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342283    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kube-rbac-proxy"
Jul 02 22:09:17.342558 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342330    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kserve-container"
Jul 02 22:09:17.342558 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342338    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="80691226-a702-4432-b4c1-08223db2b3db" containerName="kube-rbac-proxy"
Jul 02 22:09:17.342558 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342343    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kube-rbac-proxy"
Jul 02 22:09:17.342558 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342351    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af0d93e-02b1-4c46-832a-6580f162dfbb" containerName="kserve-container"
Jul 02 22:09:17.342558 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342356    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kube-rbac-proxy"
Jul 02 22:09:17.342558 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.342362    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="741d6758-2db9-420b-aaae-08ad6ff4c4e6" containerName="kserve-container"
Jul 02 22:09:17.346890 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.346856    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.352243 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.352220    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-28brx\"/\"openshift-service-ca.crt\""
Jul 02 22:09:17.352602 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.352584    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-28brx\"/\"kube-root-ca.crt\""
Jul 02 22:09:17.353478 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.353458    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-28brx\"/\"default-dockercfg-4gx7c\""
Jul 02 22:09:17.363501 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.363471    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"]
Jul 02 22:09:17.399249 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.399218    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-857f4fb655-8mdxx_905a5356-350d-49b4-8147-40238b85b1d9/console/0.log"
Jul 02 22:09:17.399496 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.399476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-proc\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.399584 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.399541    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99gv9\" (UniqueName: \"kubernetes.io/projected/23b29a42-33ba-4186-87b6-0c6eb82e2db7-kube-api-access-99gv9\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.399636 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.399586    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-lib-modules\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.399681 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.399638    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-sys\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.399726 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.399708    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-podres\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.442234 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.442189    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-z2mms_15e84cd5-bccd-4749-b3d3-2f5d805ab6e6/download-server/0.log"
Jul 02 22:09:17.500219 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.500182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-proc\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.500647 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.500239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99gv9\" (UniqueName: \"kubernetes.io/projected/23b29a42-33ba-4186-87b6-0c6eb82e2db7-kube-api-access-99gv9\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.500647 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.500275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-lib-modules\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.500647 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.500288    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-proc\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.500647 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.500291    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-sys\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.500647 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.500341    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-sys\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.500647 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.500354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-podres\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.500647 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.500450    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-podres\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.500647 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.500471    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23b29a42-33ba-4186-87b6-0c6eb82e2db7-lib-modules\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.508452 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.508426    2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99gv9\" (UniqueName: \"kubernetes.io/projected/23b29a42-33ba-4186-87b6-0c6eb82e2db7-kube-api-access-99gv9\") pod \"perf-node-gather-daemonset-mxq5n\" (UID: \"23b29a42-33ba-4186-87b6-0c6eb82e2db7\") " pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.657553 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.657438    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:17.785871 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:17.785844    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"]
Jul 02 22:09:17.788382 ip-10-0-130-121 kubenswrapper[2564]: W0702 22:09:17.788352    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23b29a42_33ba_4186_87b6_0c6eb82e2db7.slice/crio-3089a6beb2ff8b98225fc0369b79828c72721899d940a4921138f8d257ad7bb9 WatchSource:0}: Error finding container 3089a6beb2ff8b98225fc0369b79828c72721899d940a4921138f8d257ad7bb9: Status 404 returned error can't find the container with id 3089a6beb2ff8b98225fc0369b79828c72721899d940a4921138f8d257ad7bb9
Jul 02 22:09:18.603247 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:18.603203    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n" event={"ID":"23b29a42-33ba-4186-87b6-0c6eb82e2db7","Type":"ContainerStarted","Data":"94ae20d192570beec41e5a53e25e85bd28e036c3102f7efa9606c6ddf1d14f83"}
Jul 02 22:09:18.603605 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:18.603263    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n" event={"ID":"23b29a42-33ba-4186-87b6-0c6eb82e2db7","Type":"ContainerStarted","Data":"3089a6beb2ff8b98225fc0369b79828c72721899d940a4921138f8d257ad7bb9"}
Jul 02 22:09:18.603605 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:18.603289    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:18.620317 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:18.620266    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n" podStartSLOduration=1.620250955 podStartE2EDuration="1.620250955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 22:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 22:09:18.619599343 +0000 UTC m=+3027.353232661" watchObservedRunningTime="2026-07-02 22:09:18.620250955 +0000 UTC m=+3027.353884262"
Jul 02 22:09:18.869452 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:18.869355    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-948td_f32deb0f-16bc-480c-90c8-70fb7d8e81b6/dns/0.log"
Jul 02 22:09:18.893029 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:18.892978    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-948td_f32deb0f-16bc-480c-90c8-70fb7d8e81b6/kube-rbac-proxy/0.log"
Jul 02 22:09:19.046397 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:19.046366    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6st8d_d928b5bc-8033-4f03-be65-0989dabe8581/dns-node-resolver/0.log"
Jul 02 22:09:19.551141 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:19.551114    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f445984c5-q9n6l_0bebaf20-3e91-4341-b6c2-498ab54f5fdd/registry/0.log"
Jul 02 22:09:19.603428 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:19.603401    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g55b4_1c5ca752-f1cb-483c-9f16-7b425d081d7b/node-ca/0.log"
Jul 02 22:09:20.811964 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:20.811932    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6xfl8_33841b4b-1329-4435-801f-3e2f803b298e/serve-healthcheck-canary/0.log"
Jul 02 22:09:21.202471 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:21.202445    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f7sz7_a6e7e14b-5fe7-4b9d-9edc-496ceb857389/kube-rbac-proxy/0.log"
Jul 02 22:09:21.224539 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:21.224513    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f7sz7_a6e7e14b-5fe7-4b9d-9edc-496ceb857389/exporter/0.log"
Jul 02 22:09:21.247773 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:21.247742    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f7sz7_a6e7e14b-5fe7-4b9d-9edc-496ceb857389/extractor/0.log"
Jul 02 22:09:23.571770 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:23.571738    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-56668bddcf-fpqmb_3816d407-592b-43fe-9219-e516cdfc949c/manager/0.log"
Jul 02 22:09:23.618813 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:23.618747    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-h7bw7_6dca3ea7-71e8-4657-a64d-fe18841eac6f/server/0.log"
Jul 02 22:09:23.771967 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:23.771936    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-jtqwk_e833578a-9fa9-440d-a13d-197c1043b539/manager/0.log"
Jul 02 22:09:23.862332 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:23.862241    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-tfm28_8758349f-2525-492a-b49c-af4e553ec097/seaweedfs/0.log"
Jul 02 22:09:24.616573 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:24.616543    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-28brx/perf-node-gather-daemonset-mxq5n"
Jul 02 22:09:29.663613 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:29.663588    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdmdj_247b8f4d-bbc5-44b7-981b-aa42c30c031f/kube-multus-additional-cni-plugins/0.log"
Jul 02 22:09:29.686091 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:29.686026    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdmdj_247b8f4d-bbc5-44b7-981b-aa42c30c031f/egress-router-binary-copy/0.log"
Jul 02 22:09:29.707686 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:29.707665    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdmdj_247b8f4d-bbc5-44b7-981b-aa42c30c031f/cni-plugins/0.log"
Jul 02 22:09:29.729030 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:29.728973    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdmdj_247b8f4d-bbc5-44b7-981b-aa42c30c031f/bond-cni-plugin/0.log"
Jul 02 22:09:29.752911 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:29.752887    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdmdj_247b8f4d-bbc5-44b7-981b-aa42c30c031f/routeoverride-cni/0.log"
Jul 02 22:09:29.774777 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:29.774759    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdmdj_247b8f4d-bbc5-44b7-981b-aa42c30c031f/whereabouts-cni-bincopy/0.log"
Jul 02 22:09:29.798772 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:29.798751    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdmdj_247b8f4d-bbc5-44b7-981b-aa42c30c031f/whereabouts-cni/0.log"
Jul 02 22:09:29.841895 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:29.841876    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bllj5_6c7ccba7-ce5b-45ac-ba36-22db25a36607/kube-multus/0.log"
Jul 02 22:09:29.970836 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:29.970809    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fd5sb_30cc5577-2894-4468-9c39-a462c823e44e/network-metrics-daemon/0.log"
Jul 02 22:09:29.992111 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:29.992089    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fd5sb_30cc5577-2894-4468-9c39-a462c823e44e/kube-rbac-proxy/0.log"
Jul 02 22:09:30.817417 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:30.817386    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cd8lj_61608ec6-2071-4478-9bbf-4f8a55085fd9/ovn-controller/0.log"
Jul 02 22:09:30.836021 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:30.835982    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cd8lj_61608ec6-2071-4478-9bbf-4f8a55085fd9/ovn-acl-logging/0.log"
Jul 02 22:09:30.875116 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:30.875083    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cd8lj_61608ec6-2071-4478-9bbf-4f8a55085fd9/kube-rbac-proxy-node/0.log"
Jul 02 22:09:30.897569 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:30.897547    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cd8lj_61608ec6-2071-4478-9bbf-4f8a55085fd9/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 22:09:30.916797 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:30.916778    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cd8lj_61608ec6-2071-4478-9bbf-4f8a55085fd9/northd/0.log"
Jul 02 22:09:30.937203 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:30.937168    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cd8lj_61608ec6-2071-4478-9bbf-4f8a55085fd9/nbdb/0.log"
Jul 02 22:09:30.959433 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:30.959406    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cd8lj_61608ec6-2071-4478-9bbf-4f8a55085fd9/sbdb/0.log"
Jul 02 22:09:31.025022 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:31.024189    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cd8lj_61608ec6-2071-4478-9bbf-4f8a55085fd9/ovnkube-controller/0.log"
Jul 02 22:09:32.550649 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:32.550615    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-pjcpf_a723a652-5f75-4d27-acc7-5489c2aba607/network-check-target-container/0.log"
Jul 02 22:09:33.690006 ip-10-0-130-121 kubenswrapper[2564]: I0702 22:09:33.689970    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-gw9bj_178a3739-7dec-45fc-a1f0-58d7e82806ff/iptables-alerter/0.log"