Jul 24 21:14:37.767962 ip-10-0-140-162 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 21:14:38.214831 ip-10-0-140-162 kubenswrapper[2522]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:14:38.214831 ip-10-0-140-162 kubenswrapper[2522]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 21:14:38.214831 ip-10-0-140-162 kubenswrapper[2522]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:14:38.214831 ip-10-0-140-162 kubenswrapper[2522]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 21:14:38.214831 ip-10-0-140-162 kubenswrapper[2522]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:14:38.215775 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.215696    2522 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 21:14:38.220446 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220431    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:14:38.220446 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220446    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220450    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220456    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220460    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220463    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220466    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220470    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220473    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220476    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220478    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220481    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220484    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220486    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220489    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220491    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220494    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220496    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220499    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220501    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220504    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:14:38.220508 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220506    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220509    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220511    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220514    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220517    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220520    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220522    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220528    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220531    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220533    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220536    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220538    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220541    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220543    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220546    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220548    2522 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220551    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220554    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220558    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:14:38.220996 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220561    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220564    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220566    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220569    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220571    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220573    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220577    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220579    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220581    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220584    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220586    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220589    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220592    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220594    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220596    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220600    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220603    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220605    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220608    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220610    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:14:38.221482 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220613    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220615    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220618    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220620    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220623    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220625    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220628    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220630    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220633    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220635    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220639    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220642    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220645    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220648    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220652    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220654    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220657    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220659    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220662    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:14:38.221980 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220665    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220681    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220684    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220686    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220689    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220692    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220694    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220698    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220701    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220704    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220707    2522 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220709    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220712    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220714    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220717    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220719    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220722    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220725    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220727    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.220730    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:14:38.222458 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221404    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221412    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221415    2522 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221417    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221420    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221422    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221425    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221428    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221430    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221433    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221435    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221438    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221440    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221458    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221461    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221463    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221466    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221468    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221471    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221475    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:14:38.222937 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221477    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221480    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221482    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221485    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221487    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221490    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221493    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221496    2522 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221498    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221501    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221503    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221506    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221509    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221514    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221517    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221520    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221523    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221525    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221528    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:14:38.223420 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221530    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221533    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221535    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221538    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221541    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221543    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221546    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221549    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221551    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221555    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221559    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221562    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221565    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221568    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221570    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221573    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221575    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221578    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221580    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:14:38.223903 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221583    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221585    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221588    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221590    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221594    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221597    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221600    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221603    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221606    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221609    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221612    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221614    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221617    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221620    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221623    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221625    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221628    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221631    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221633    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221636    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:14:38.224355 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221638    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221641    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221645    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221648    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221683    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221688    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221691    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221694    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221696    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221699    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221702    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221704    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221707    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221710    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221712    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221714    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221717    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221719    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221722    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221726    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:14:38.224836 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.221729    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222082    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222089    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222091    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222095    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222097    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222100    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222103    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222105    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222108    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222110    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222113    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222115    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222118    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222121    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222124    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222130    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222133    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222136    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:14:38.225305 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222139    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222141    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222144    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222146    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222149    2522 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222151    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222154    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222158    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222161    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222164    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222167    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222170    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222173    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222175    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222178    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222183    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222185    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222188    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222190    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222193    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:14:38.225787 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222195    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222198    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222200    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222203    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222206    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222208    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222211    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222214    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222217    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222221    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222224    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222226    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222228    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222231    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222233    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222236    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222238    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222241    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222243    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222246    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:14:38.226281 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222248    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222251    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222253    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222256    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222258    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222261    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222263    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222267    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222269    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222272    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222274    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222277    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222279    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222282    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222284    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222286    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222289    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222291    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222294    2522 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222296    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:14:38.226767 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222299    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222304    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222307    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222311    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222313    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222316    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222318    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222321    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222323    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222325    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222327    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222330    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222333    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222335    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222338    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222340    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222342    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222345    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222349    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:14:38.227241 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222354    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.222357    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222427    2522 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222433    2522 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222440    2522 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222444    2522 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222448    2522 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222451    2522 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222454    2522 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222458    2522 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222462    2522 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222465    2522 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222468    2522 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222471    2522 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222483    2522 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222486    2522 flags.go:64] FLAG: --cgroup-root=""
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222489    2522 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222492    2522 flags.go:64] FLAG: --client-ca-file=""
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222494    2522 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222497    2522 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222504    2522 flags.go:64] FLAG: --cluster-domain=""
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222506    2522 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222509    2522 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222512    2522 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 21:14:38.227745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222516    2522 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222519    2522 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222522    2522 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222525    2522 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222528    2522 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222531    2522 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222534    2522 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222537    2522 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222541    2522 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222543    2522 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222548    2522 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222551    2522 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222553    2522 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222556    2522 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222559    2522 flags.go:64] FLAG: --enable-server="true"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222561    2522 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222568    2522 flags.go:64] FLAG: --event-burst="100"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222571    2522 flags.go:64] FLAG: --event-qps="50"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222574    2522 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222577    2522 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222579    2522 flags.go:64] FLAG: --eviction-hard=""
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222583    2522 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222586    2522 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222589    2522 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222592    2522 flags.go:64] FLAG: --eviction-soft=""
Jul 24 21:14:38.228312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222595    2522 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222597    2522 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222600    2522 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222603    2522 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222606    2522 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222609    2522 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222611    2522 flags.go:64] FLAG: --feature-gates=""
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222615    2522 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222617    2522 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222620    2522 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222623    2522 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222626    2522 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222629    2522 flags.go:64] FLAG: --help="false"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222632    2522 flags.go:64] FLAG: --hostname-override="ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222634    2522 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222638    2522 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222641    2522 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222644    2522 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222648    2522 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222651    2522 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222653    2522 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222656    2522 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222658    2522 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 21:14:38.228917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222661    2522 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222664    2522 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222667    2522 flags.go:64] FLAG: --kube-reserved=""
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222685    2522 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222688    2522 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222691    2522 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222694    2522 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222697    2522 flags.go:64] FLAG: --lock-file=""
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222700    2522 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222703    2522 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222706    2522 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222711    2522 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222714    2522 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222716    2522 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222719    2522 flags.go:64] FLAG: --logging-format="text"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222722    2522 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222725    2522 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222728    2522 flags.go:64] FLAG: --manifest-url=""
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222730    2522 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222734    2522 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222737    2522 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222741    2522 flags.go:64] FLAG: --max-pods="110"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222744    2522 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222747    2522 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222750    2522 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 21:14:38.229509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222754    2522 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222757    2522 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222759    2522 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222762    2522 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222772    2522 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222775    2522 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222778    2522 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222781    2522 flags.go:64] FLAG: --pod-cidr=""
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222784    2522 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222789    2522 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222792    2522 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222795    2522 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222798    2522 flags.go:64] FLAG: --port="10250"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222800    2522 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222803    2522 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dde57d5255c07ee1"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222806    2522 flags.go:64] FLAG: --qos-reserved=""
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222810    2522 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222812    2522 flags.go:64] FLAG: --register-node="true"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222815    2522 flags.go:64] FLAG: --register-with-taints=""
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222818    2522 flags.go:64] FLAG: --registry-burst="10"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222821    2522 flags.go:64] FLAG: --registry-qps="5"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222824    2522 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222827    2522 flags.go:64] FLAG: --reserved-memory=""
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222830    2522 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222833    2522 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 21:14:38.230223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222836    2522 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222838    2522 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222841    2522 flags.go:64] FLAG: --runonce="false"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222844    2522 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222847    2522 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222849    2522 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222852    2522 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222856    2522 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222859    2522 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222862    2522 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222865    2522 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222867    2522 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222870    2522 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222874    2522 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222876    2522 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222879    2522 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222882    2522 flags.go:64] FLAG: --system-cgroups=""
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222884    2522 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222889    2522 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222893    2522 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222896    2522 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222902    2522 flags.go:64] FLAG: --tls-min-version=""
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222905    2522 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222907    2522 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222910    2522 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 21:14:38.230844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222913    2522 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222916    2522 flags.go:64] FLAG: --v="2"
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222919    2522 flags.go:64] FLAG: --version="false"
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222923    2522 flags.go:64] FLAG: --vmodule=""
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222927    2522 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.222930    2522 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223079    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223084    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223087    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223090    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223092    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223095    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223097    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223100    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223103    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223106    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223108    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223110    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223113    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223116    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223118    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:14:38.231488 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223120    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223123    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223125    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223128    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223131    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223137    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223140    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223142    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223145    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223148    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223150    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223153    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223156    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223159    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223161    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223164    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223166    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223169    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223172    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:14:38.232005 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223175    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223177    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223179    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223182    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223184    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223187    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223189    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223192    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223194    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223197    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223199    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223202    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223204    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223207    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223209    2522 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223212    2522 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223214    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223216    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223220    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:14:38.232459 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223223    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223226    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223230    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223233    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223236    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223238    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223242    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223244    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223246    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223249    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223251    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223253    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223256    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223258    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223260    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223263    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223266    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223270    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223272    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:14:38.232969 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223275    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223277    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223279    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223282    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223284    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223287    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223289    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223291    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223294    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223296    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223298    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223301    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223304    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223307    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223309    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223311    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223314    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223316    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223319    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223321    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:14:38.233434 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223324    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223326    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223328    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223331    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223333    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223336    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.223338    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.223867    2522 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.230482    2522 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.230496    2522 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.230515    2522 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.230521    2522 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230594    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230600    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:14:38.233946 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230603    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230605    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230608    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230611    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230614    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230616    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230619    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230621    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230624    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230626    2522 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230629    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230631    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230634    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230636    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230639    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230641    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230644    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230646    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230649    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230652    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:14:38.234296 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230654    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230656    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230659    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230661    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230663    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230667    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230688    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230693    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230696    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230699    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230702    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230705    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230707    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230710    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230712    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230715    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230717    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230720    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230722    2522 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230725    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:14:38.234794 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230728    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230731    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230735    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230739    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230741    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230744    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230747    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230749    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230752    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230754    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230757    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230759    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230762    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230765    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230767    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230769    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230772    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230775    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230778    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:14:38.235270 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230782    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230786    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230789    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230792    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230795    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230797    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230800    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230802    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230805    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230807    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230810    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230812    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230815    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230817    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230820    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230822    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230824    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230827    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230829    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230832    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:14:38.235736 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230834    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230837    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230839    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230842    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230844    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230847    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230849    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230852    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230856    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230859    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230862    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230865    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230868    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230871    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230875    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230877    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230880    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:14:38.236216 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.230883    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.230887    2522 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231001    2522 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231005    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231008    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231011    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231014    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231017    2522 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231020    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231022    2522 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231025    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231028    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231031    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231033    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231035    2522 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:14:38.236628 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231038    2522 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231040    2522 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231043    2522 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231045    2522 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231048    2522 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231050    2522 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231052    2522 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231055    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231057    2522 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231060    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231062    2522 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231065    2522 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231068    2522 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231070    2522 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231073    2522 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231076    2522 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231078    2522 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231080    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231083    2522 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231085    2522 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:14:38.237150 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231088    2522 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231090    2522 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231092    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231095    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231097    2522 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231099    2522 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231102    2522 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231105    2522 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231107    2522 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231109    2522 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231112    2522 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231114    2522 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231118    2522 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231122    2522 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231125    2522 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231128    2522 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231130    2522 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231133    2522 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231136    2522 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:14:38.237643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231138    2522 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231141    2522 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231143    2522 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231146    2522 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231149    2522 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231152    2522 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231154    2522 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231156    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231160    2522 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231164    2522 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231166    2522 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231169    2522 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231171    2522 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231173    2522 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231177    2522 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231180    2522 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231183    2522 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231186    2522 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231188    2522 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:14:38.238126 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231191    2522 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231193    2522 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231196    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231198    2522 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231201    2522 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231203    2522 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231205    2522 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231208    2522 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231210    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231212    2522 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231215    2522 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231217    2522 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231219    2522 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231222    2522 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231224    2522 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231226    2522 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231229    2522 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231231    2522 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231234    2522 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231236    2522 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:14:38.238572 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231238    2522 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231241    2522 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231243    2522 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231246    2522 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231248    2522 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231251    2522 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231253    2522 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:38.231256    2522 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.231260    2522 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.231356    2522 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.233294    2522 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.234453    2522 server.go:1019] "Starting client certificate rotation"
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.234551    2522 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 21:14:38.239064 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.234585    2522 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 21:14:38.239419 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.236720    2522 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 21:14:38.239419 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.239330    2522 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 21:14:38.248911 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.248892    2522 log.go:25] "Validated CRI v1 runtime API"
Jul 24 21:14:38.250880 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.250863    2522 log.go:25] "Validated CRI v1 image API"
Jul 24 21:14:38.251631 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.251615    2522 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 21:14:38.253936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.253917    2522 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ce76cf12-e33d-4dea-8d83-c5fbecf467bc:/dev/nvme0n1p3 d8ad7fd0-068e-40c8-9b04-34ced980389c:/dev/nvme0n1p4]
Jul 24 21:14:38.254004 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.253935    2522 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 21:14:38.259780 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.259656    2522 manager.go:216] Machine: {Timestamp:2026-07-24 21:14:38.258349375 +0000 UTC m=+0.289744374 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3139170 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23ac98a840f3c94a34d0c549aa9479 SystemUUID:ec23ac98-a840-f3c9-4a34-d0c549aa9479 BootID:d35c337d-347a-408a-8848-e677f6a1bf8e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:32:f5:2d:f9:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:32:f5:2d:f9:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:16:f9:7d:b8:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 21:14:38.259780 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.259774    2522 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 21:14:38.259882 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.259871    2522 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 21:14:38.262439 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.262413    2522 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 21:14:38.262569 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.262443    2522 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-162.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 21:14:38.263070 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.263060    2522 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 21:14:38.263109 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.263073    2522 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 21:14:38.263109 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.263085    2522 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 21:14:38.263109 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.263100    2522 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 21:14:38.263812 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.263802    2522 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 21:14:38.265454 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.265442    2522 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 21:14:38.265578 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.265569    2522 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 21:14:38.266448 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.266435    2522 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 21:14:38.266481 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.266458    2522 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 21:14:38.266481 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.266474    2522 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 21:14:38.266549 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.266485    2522 kubelet.go:398] "Adding apiserver pod source"
Jul 24 21:14:38.266549 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.266497    2522 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 21:14:38.266898 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.266886    2522 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 21:14:38.266938 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.266906    2522 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 21:14:38.268330 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.268311    2522 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 21:14:38.269056 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.269037    2522 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 21:14:38.271145 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.271131    2522 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 21:14:38.271191 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.271161    2522 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 21:14:38.272414 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272403    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 21:14:38.272452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272418    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 21:14:38.272452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272425    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 21:14:38.272452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272430    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 21:14:38.272452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272438    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 21:14:38.272452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272444    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 21:14:38.272452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272452    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 21:14:38.272604 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272457    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 21:14:38.272604 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272467    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 21:14:38.272604 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272472    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 21:14:38.272604 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272480    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 21:14:38.272604 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.272485    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 21:14:38.273178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.273166    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 21:14:38.273207 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.273187    2522 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 21:14:38.276605 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.276592    2522 server.go:1268] "Started kubelet"
Jul 24 21:14:38.276708 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.276666    2522 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 21:14:38.277014 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.276988    2522 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-162.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 21:14:38.277014 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.276994    2522 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 21:14:38.277154 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.277106    2522 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-162.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 21:14:38.277511 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.277450    2522 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 21:14:38.277591 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.277526    2522 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 21:14:38.277625 ip-10-0-140-162 systemd[1]: Started Kubernetes Kubelet.
Jul 24 21:14:38.281795 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.279130    2522 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 21:14:38.282609 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.282583    2522 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 21:14:38.282778 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.282758    2522 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 21:14:38.287690 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.286720    2522 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-162.ec2.internal.18c55644356f928f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-162.ec2.internal,UID:ip-10-0-140-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-162.ec2.internal,},FirstTimestamp:2026-07-24 21:14:38.276539023 +0000 UTC m=+0.307934027,LastTimestamp:2026-07-24 21:14:38.276539023 +0000 UTC m=+0.307934027,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-162.ec2.internal,}"
Jul 24 21:14:38.288221 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.288200    2522 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 21:14:38.288297 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.288222    2522 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 21:14:38.288544 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.288525    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:38.288932 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.288861    2522 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 21:14:38.289017 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289004    2522 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 21:14:38.289070 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289018    2522 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 21:14:38.289120 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289107    2522 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 21:14:38.289120 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289118    2522 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 21:14:38.289201 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289180    2522 factory.go:55] Registering systemd factory
Jul 24 21:14:38.289201 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289193    2522 factory.go:223] Registration of the systemd container factory successfully
Jul 24 21:14:38.289416 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289400    2522 factory.go:153] Registering CRI-O factory
Jul 24 21:14:38.289519 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289421    2522 factory.go:223] Registration of the crio container factory successfully
Jul 24 21:14:38.289519 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289468    2522 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 21:14:38.289519 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289489    2522 factory.go:103] Registering Raw factory
Jul 24 21:14:38.289519 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289499    2522 manager.go:1196] Started watching for new ooms in manager
Jul 24 21:14:38.289839 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.289821    2522 manager.go:349] Starting recovery of all containers
Jul 24 21:14:38.293055 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.293027    2522 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-162.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 21:14:38.293398 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.293358    2522 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 21:14:38.297560 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.297533    2522 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 21:14:38.301309 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.301294    2522 manager.go:354] Recovery completed
Jul 24 21:14:38.305349 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.305337    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:14:38.308111 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.308085    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:14:38.308151 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.308125    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:14:38.308151 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.308137    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:14:38.309119 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.309105    2522 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 21:14:38.309185 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.309120    2522 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 21:14:38.309185 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.309130    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:14:38.309185 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.309137    2522 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 21:14:38.310053 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.310034    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:14:38.310134 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.310064    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:14:38.310134 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.310078    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:14:38.310731 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.310609    2522 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-162.ec2.internal.18c556443751568b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-162.ec2.internal,UID:ip-10-0-140-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-162.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-162.ec2.internal,},FirstTimestamp:2026-07-24 21:14:38.308112011 +0000 UTC m=+0.339507010,LastTimestamp:2026-07-24 21:14:38.308112011 +0000 UTC m=+0.339507010,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-162.ec2.internal,}"
Jul 24 21:14:38.310817 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.310769    2522 policy_none.go:49] "None policy: Start"
Jul 24 21:14:38.310817 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.310784    2522 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 21:14:38.310817 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.310794    2522 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 21:14:38.311752 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.311740    2522 policy_none.go:47] "Start"
Jul 24 21:14:38.319801 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.319741    2522 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-162.ec2.internal.18c556443751a549  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-162.ec2.internal,UID:ip-10-0-140-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-162.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-162.ec2.internal,},FirstTimestamp:2026-07-24 21:14:38.308132169 +0000 UTC m=+0.339527168,LastTimestamp:2026-07-24 21:14:38.308132169 +0000 UTC m=+0.339527168,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-162.ec2.internal,}"
Jul 24 21:14:38.322784 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.322734    2522 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-162.ec2.internal.18c556443751c6e9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-162.ec2.internal,UID:ip-10-0-140-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-162.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-162.ec2.internal,},FirstTimestamp:2026-07-24 21:14:38.308140777 +0000 UTC m=+0.339535777,LastTimestamp:2026-07-24 21:14:38.308140777 +0000 UTC m=+0.339535777,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-162.ec2.internal,}"
Jul 24 21:14:38.325770 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.325755    2522 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c49jp"
Jul 24 21:14:38.330376 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.330214    2522 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-162.ec2.internal.18c556443751568b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-162.ec2.internal.18c556443751568b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-162.ec2.internal,UID:ip-10-0-140-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-162.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-162.ec2.internal,},FirstTimestamp:2026-07-24 21:14:38.308112011 +0000 UTC m=+0.339507010,LastTimestamp:2026-07-24 21:14:38.310049491 +0000 UTC m=+0.341444495,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-162.ec2.internal,}"
Jul 24 21:14:38.332145 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.332083    2522 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-162.ec2.internal.18c556443751a549\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-162.ec2.internal.18c556443751a549  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-162.ec2.internal,UID:ip-10-0-140-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-162.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-162.ec2.internal,},FirstTimestamp:2026-07-24 21:14:38.308132169 +0000 UTC m=+0.339527168,LastTimestamp:2026-07-24 21:14:38.310071211 +0000 UTC m=+0.341466212,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-162.ec2.internal,}"
Jul 24 21:14:38.333561 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.333543    2522 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c49jp"
Jul 24 21:14:38.353247 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.353231    2522 manager.go:337] "Starting Device Plugin manager"
Jul 24 21:14:38.369285 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.353272    2522 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 21:14:38.369285 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.353286    2522 server.go:87] "Starting device plugin registration server"
Jul 24 21:14:38.369285 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.353506    2522 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 21:14:38.369285 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.353521    2522 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 21:14:38.369285 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.353615    2522 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 21:14:38.369285 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.353709    2522 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 21:14:38.369285 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.353722    2522 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 21:14:38.369285 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.355106    2522 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 21:14:38.369285 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.355155    2522 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:38.400821 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.400798    2522 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 21:14:38.400900 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.400837    2522 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 21:14:38.400900 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.400854    2522 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 21:14:38.400900 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.400883    2522 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 21:14:38.403340 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.403327    2522 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:14:38.454435 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.454419    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:14:38.455204 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.455188    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:14:38.455268 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.455223    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:14:38.455268 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.455236    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:14:38.455268 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.455258    2522 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.465238 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.465200    2522 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.465238 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.465220    2522 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-162.ec2.internal\": node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:38.485281 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.485262    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:38.501612 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.501585    2522 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-162.ec2.internal"]
Jul 24 21:14:38.501663 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.501646    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:14:38.502409 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.502396    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:14:38.502455 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.502419    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:14:38.502455 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.502429    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:14:38.504748 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.504735    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:14:38.504888 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.504874    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.504934 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.504901    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:14:38.505460 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.505442    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:14:38.505540 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.505475    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:14:38.505540 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.505490    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:14:38.505540 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.505442    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:14:38.505707 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.505559    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:14:38.505707 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.505571    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:14:38.507829 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.507812    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.507919 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.507841    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:14:38.508562 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.508545    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:14:38.508645 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.508578    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:14:38.508645 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.508590    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:14:38.522204 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.522186    2522 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-162.ec2.internal\" not found" node="ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.526552 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.526535    2522 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-162.ec2.internal\" not found" node="ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.585320 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.585299    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:38.590717 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.590701    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66217b0fb27a8d11ff5105a5ffa34535-config\") pod \"kube-apiserver-proxy-ip-10-0-140-162.ec2.internal\" (UID: \"66217b0fb27a8d11ff5105a5ffa34535\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.590775 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.590724    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d5eee1a56b131b18e3bafe96ff6f390-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal\" (UID: \"6d5eee1a56b131b18e3bafe96ff6f390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.590775 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.590742    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d5eee1a56b131b18e3bafe96ff6f390-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal\" (UID: \"6d5eee1a56b131b18e3bafe96ff6f390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.685624 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.685603    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:38.690940 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.690924    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d5eee1a56b131b18e3bafe96ff6f390-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal\" (UID: \"6d5eee1a56b131b18e3bafe96ff6f390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.690940 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.690933    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d5eee1a56b131b18e3bafe96ff6f390-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal\" (UID: \"6d5eee1a56b131b18e3bafe96ff6f390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.691034 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.690960    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d5eee1a56b131b18e3bafe96ff6f390-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal\" (UID: \"6d5eee1a56b131b18e3bafe96ff6f390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.691034 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.690983    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66217b0fb27a8d11ff5105a5ffa34535-config\") pod \"kube-apiserver-proxy-ip-10-0-140-162.ec2.internal\" (UID: \"66217b0fb27a8d11ff5105a5ffa34535\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.691034 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.691018    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66217b0fb27a8d11ff5105a5ffa34535-config\") pod \"kube-apiserver-proxy-ip-10-0-140-162.ec2.internal\" (UID: \"66217b0fb27a8d11ff5105a5ffa34535\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.691135 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.691050    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d5eee1a56b131b18e3bafe96ff6f390-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal\" (UID: \"6d5eee1a56b131b18e3bafe96ff6f390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.786261 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.786209    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:38.824664 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.824646    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.829361 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:38.829205    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:38.887115 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.887088    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:38.987605 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:38.987579    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:39.088079 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:39.088031    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:39.188581 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:39.188561    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:39.233973 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.233954    2522 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 21:14:39.234340 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.234086    2522 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 21:14:39.288717 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:39.288693    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:39.338021 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.337991    2522 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 21:09:39 +0000 UTC" deadline="2028-03-19 12:43:01.73830598 +0000 UTC"
Jul 24 21:14:39.338021 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.338020    2522 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14487h28m22.400288957s"
Jul 24 21:14:39.389346 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:39.389326    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:39.397523 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:39.397499    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5eee1a56b131b18e3bafe96ff6f390.slice/crio-0bdc5e23bef428288bd8d0cfcc9e0e71252be15f12f39c7db814e382eed0eec7 WatchSource:0}: Error finding container 0bdc5e23bef428288bd8d0cfcc9e0e71252be15f12f39c7db814e382eed0eec7: Status 404 returned error can't find the container with id 0bdc5e23bef428288bd8d0cfcc9e0e71252be15f12f39c7db814e382eed0eec7
Jul 24 21:14:39.399410 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:39.399383    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66217b0fb27a8d11ff5105a5ffa34535.slice/crio-48ec4233734a7997ad4a26259790963f20771f4cc5081c6473fb987a38fa0c92 WatchSource:0}: Error finding container 48ec4233734a7997ad4a26259790963f20771f4cc5081c6473fb987a38fa0c92: Status 404 returned error can't find the container with id 48ec4233734a7997ad4a26259790963f20771f4cc5081c6473fb987a38fa0c92
Jul 24 21:14:39.404253 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.404236    2522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:14:39.471144 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.471123    2522 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:14:39.472188 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.472173    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:14:39.472273 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.472202    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:14:39.472273 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.472212    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:14:39.489455 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:39.489439    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:39.494821 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.494804    2522 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:14:39.589820 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:39.589793    2522 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-162.ec2.internal\" not found"
Jul 24 21:14:39.630426 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.630403    2522 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:14:39.664709 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.664646    2522 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:14:39.689682 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.689656    2522 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:39.703084 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.703062    2522 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 21:14:39.704072 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.704059    2522 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-162.ec2.internal"
Jul 24 21:14:39.710136 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:39.710122    2522 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 21:14:40.099284 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.099257    2522 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:14:40.267722 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.267692    2522 apiserver.go:52] "Watching apiserver"
Jul 24 21:14:40.273315 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.273287    2522 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 21:14:40.279541 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.279506    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-ntvd4","kube-system/global-pull-secret-syncer-sxqs9","kube-system/kube-apiserver-proxy-ip-10-0-140-162.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz","openshift-dns/node-resolver-sw9pw","openshift-image-registry/node-ca-zsccb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal","openshift-multus/multus-additional-cni-plugins-xcw5h","openshift-multus/network-metrics-daemon-krmvj","kube-system/konnectivity-agent-gfsnb","openshift-cluster-node-tuning-operator/tuned-xdzf5","openshift-multus/multus-r57zx","openshift-network-diagnostics/network-check-target-9nzl9","openshift-network-operator/iptables-alerter-xlrn4"]
Jul 24 21:14:40.282097 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.282072    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:14:40.284207 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.284187    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:40.284313 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.284255    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:14:40.284609 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.284585    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 21:14:40.284881 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.284735    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 21:14:40.284881 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.284808    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t8pgs\""
Jul 24 21:14:40.286452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.286431    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xlrn4"
Jul 24 21:14:40.288656 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.288639    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4smmh\""
Jul 24 21:14:40.288762 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.288736    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:14:40.288762 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.288637    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.288924 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.288904    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 21:14:40.289207 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.289189    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 21:14:40.289287 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.289275    2522 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 21:14:40.290830 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.290811    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 21:14:40.290931 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.290914    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sw9pw"
Jul 24 21:14:40.291280 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.291260    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 21:14:40.291364 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.291322    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mlzpv\""
Jul 24 21:14:40.291426 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.291323    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 21:14:40.292793 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.292778    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 21:14:40.293401 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.293069    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 21:14:40.293401 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.293127    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fg6bz\""
Jul 24 21:14:40.293401 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.293221    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zsccb"
Jul 24 21:14:40.295792 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.295455    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 21:14:40.295792 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.295543    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:40.295792 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.295601    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:14:40.295792 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.295664    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 21:14:40.295792 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.295664    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qc5z8\""
Jul 24 21:14:40.296033 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.295873    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 21:14:40.297880 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.297859    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.299118 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.298929    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c62472f-b8c0-4ea0-9d99-5703ddf06036-host-slash\") pod \"iptables-alerter-xlrn4\" (UID: \"2c62472f-b8c0-4ea0-9d99-5703ddf06036\") " pod="openshift-network-operator/iptables-alerter-xlrn4"
Jul 24 21:14:40.299335 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299317    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-device-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.299413 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299383    2522 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 21:14:40.299413 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299384    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-etc-selinux\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.299518 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299446    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.299518 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299475    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-registration-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.299604 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299499    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-sys-fs\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.299604 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299539    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjfz5\" (UniqueName: \"kubernetes.io/projected/a6460eef-e05a-418d-9c44-36d5d9ac9867-kube-api-access-qjfz5\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.299604 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299577    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09ad0063-0c05-496f-9696-cec68aa2aa14-hosts-file\") pod \"node-resolver-sw9pw\" (UID: \"09ad0063-0c05-496f-9696-cec68aa2aa14\") " pod="openshift-dns/node-resolver-sw9pw"
Jul 24 21:14:40.299785 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299603    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sglz9\" (UniqueName: \"kubernetes.io/projected/09ad0063-0c05-496f-9696-cec68aa2aa14-kube-api-access-sglz9\") pod \"node-resolver-sw9pw\" (UID: \"09ad0063-0c05-496f-9696-cec68aa2aa14\") " pod="openshift-dns/node-resolver-sw9pw"
Jul 24 21:14:40.299785 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299691    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9vj\" (UniqueName: \"kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj\") pod \"network-check-target-9nzl9\" (UID: \"05180a7b-cf89-498d-af4b-6ba6ed49ae0e\") " pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:40.299785 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299721    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c62472f-b8c0-4ea0-9d99-5703ddf06036-iptables-alerter-script\") pod \"iptables-alerter-xlrn4\" (UID: \"2c62472f-b8c0-4ea0-9d99-5703ddf06036\") " pod="openshift-network-operator/iptables-alerter-xlrn4"
Jul 24 21:14:40.299785 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299755    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755fw\" (UniqueName: \"kubernetes.io/projected/2c62472f-b8c0-4ea0-9d99-5703ddf06036-kube-api-access-755fw\") pod \"iptables-alerter-xlrn4\" (UID: \"2c62472f-b8c0-4ea0-9d99-5703ddf06036\") " pod="openshift-network-operator/iptables-alerter-xlrn4"
Jul 24 21:14:40.299785 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299778    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-socket-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.299979 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299822    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ad0063-0c05-496f-9696-cec68aa2aa14-tmp-dir\") pod \"node-resolver-sw9pw\" (UID: \"09ad0063-0c05-496f-9696-cec68aa2aa14\") " pod="openshift-dns/node-resolver-sw9pw"
Jul 24 21:14:40.299979 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299854    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6159f1d7-ac4d-4c75-a777-636040b75350-agent-certs\") pod \"konnectivity-agent-gfsnb\" (UID: \"6159f1d7-ac4d-4c75-a777-636040b75350\") " pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:14:40.299979 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299881    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6159f1d7-ac4d-4c75-a777-636040b75350-konnectivity-ca\") pod \"konnectivity-agent-gfsnb\" (UID: \"6159f1d7-ac4d-4c75-a777-636040b75350\") " pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:14:40.299979 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.299957    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 21:14:40.300514 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.300494    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 21:14:40.300514 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.300507    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 21:14:40.300657 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.300565    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 21:14:40.300657 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.300495    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 21:14:40.300854 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.300839    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:40.300923 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.300901    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:14:40.301037 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.301021    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nj8nx\""
Jul 24 21:14:40.303692 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.303339    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.307303 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.305926    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.307303 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.305954    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 21:14:40.307303 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.305964    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 21:14:40.307303 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.306026    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kll2l\""
Jul 24 21:14:40.307303 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.306211    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 21:14:40.307303 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.306272    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 21:14:40.307303 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.306379    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 21:14:40.307303 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.306761    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 21:14:40.308376 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.308358    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.310359 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.309819    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 21:14:40.310359 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.310191    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vwb2d\""
Jul 24 21:14:40.310487 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.310402    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:14:40.311441 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.310971    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hp4xl\""
Jul 24 21:14:40.313719 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.313654    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 21:14:40.390280 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.390254    2522 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 21:14:40.400818 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400793    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-sys-fs\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.400906 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400826    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-run-ovn\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.400906 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400843    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-conf-dir\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.400906 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400857    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-etc-kubernetes\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.400906 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400878    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f320b45-0781-45f5-916d-5ac2fe9fa951-ovn-node-metrics-cert\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.400906 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400900    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-system-cni-dir\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.401054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400917    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-kubernetes\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.401054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400933    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6159f1d7-ac4d-4c75-a777-636040b75350-agent-certs\") pod \"konnectivity-agent-gfsnb\" (UID: \"6159f1d7-ac4d-4c75-a777-636040b75350\") " pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:14:40.401054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400951    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6159f1d7-ac4d-4c75-a777-636040b75350-konnectivity-ca\") pod \"konnectivity-agent-gfsnb\" (UID: \"6159f1d7-ac4d-4c75-a777-636040b75350\") " pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:14:40.401054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400968    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-systemd-units\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.401054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400983    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-var-lib-kubelet\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.401054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.400997    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-device-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.401054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401012    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-etc-selinux\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.401054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401031    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-kubelet\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.401054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401052    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/523501ee-3e85-408e-9dac-a8cee654bc6a-cni-binary-copy\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401067    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-var-lib-cni-bin\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401083    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfm67\" (UniqueName: \"kubernetes.io/projected/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-kube-api-access-kfm67\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401102    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/86301808-2ce5-4813-b527-98b3967b4d15-dbus\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401124    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-lib-modules\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401139    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjfz5\" (UniqueName: \"kubernetes.io/projected/a6460eef-e05a-418d-9c44-36d5d9ac9867-kube-api-access-qjfz5\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401155    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sglz9\" (UniqueName: \"kubernetes.io/projected/09ad0063-0c05-496f-9696-cec68aa2aa14-kube-api-access-sglz9\") pod \"node-resolver-sw9pw\" (UID: \"09ad0063-0c05-496f-9696-cec68aa2aa14\") " pod="openshift-dns/node-resolver-sw9pw"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401168    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-node-log\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401186    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-sysctl-conf\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401207    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-tuned\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401222    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401236    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-slash\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401255    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-run-systemd\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401269    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-cni-netd\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401283    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-daemon-config\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.401295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401298    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-run\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401312    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ad0063-0c05-496f-9696-cec68aa2aa14-tmp-dir\") pod \"node-resolver-sw9pw\" (UID: \"09ad0063-0c05-496f-9696-cec68aa2aa14\") " pod="openshift-dns/node-resolver-sw9pw"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401326    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7a9829f-33e2-4b02-a05a-35f9765d9fbb-host\") pod \"node-ca-zsccb\" (UID: \"b7a9829f-33e2-4b02-a05a-35f9765d9fbb\") " pod="openshift-image-registry/node-ca-zsccb"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401340    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-log-socket\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401353    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-modprobe-d\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401369    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-sysconfig\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401391    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8z5\" (UniqueName: \"kubernetes.io/projected/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-kube-api-access-fk8z5\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401405    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c62472f-b8c0-4ea0-9d99-5703ddf06036-host-slash\") pod \"iptables-alerter-xlrn4\" (UID: \"2c62472f-b8c0-4ea0-9d99-5703ddf06036\") " pod="openshift-network-operator/iptables-alerter-xlrn4"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401420    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-var-lib-openvswitch\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401437    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-run-k8s-cni-cncf-io\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401451    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-run-multus-certs\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401465    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401481    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401505    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-registration-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401519    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09ad0063-0c05-496f-9696-cec68aa2aa14-hosts-file\") pod \"node-resolver-sw9pw\" (UID: \"09ad0063-0c05-496f-9696-cec68aa2aa14\") " pod="openshift-dns/node-resolver-sw9pw"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401534    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-run-ovn-kubernetes\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401562    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-os-release\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.401746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401577    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401596    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c62472f-b8c0-4ea0-9d99-5703ddf06036-iptables-alerter-script\") pod \"iptables-alerter-xlrn4\" (UID: \"2c62472f-b8c0-4ea0-9d99-5703ddf06036\") " pod="openshift-network-operator/iptables-alerter-xlrn4"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401611    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755fw\" (UniqueName: \"kubernetes.io/projected/2c62472f-b8c0-4ea0-9d99-5703ddf06036-kube-api-access-755fw\") pod \"iptables-alerter-xlrn4\" (UID: \"2c62472f-b8c0-4ea0-9d99-5703ddf06036\") " pod="openshift-network-operator/iptables-alerter-xlrn4"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401626    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-system-cni-dir\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401646    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-os-release\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401661    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcmlg\" (UniqueName: \"kubernetes.io/projected/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-kube-api-access-jcmlg\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401695    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-cnibin\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401717    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401760    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f320b45-0781-45f5-916d-5ac2fe9fa951-env-overrides\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.401949    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09ad0063-0c05-496f-9696-cec68aa2aa14-hosts-file\") pod \"node-resolver-sw9pw\" (UID: \"09ad0063-0c05-496f-9696-cec68aa2aa14\") " pod="openshift-dns/node-resolver-sw9pw"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.402019    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-etc-selinux\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.402227    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ad0063-0c05-496f-9696-cec68aa2aa14-tmp-dir\") pod \"node-resolver-sw9pw\" (UID: \"09ad0063-0c05-496f-9696-cec68aa2aa14\") " pod="openshift-dns/node-resolver-sw9pw"
Jul 24 21:14:40.402544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.402226    2522 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 21:14:40.403129 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.402892    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c62472f-b8c0-4ea0-9d99-5703ddf06036-iptables-alerter-script\") pod \"iptables-alerter-xlrn4\" (UID: \"2c62472f-b8c0-4ea0-9d99-5703ddf06036\") " pod="openshift-network-operator/iptables-alerter-xlrn4"
Jul 24 21:14:40.403129 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403112    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6159f1d7-ac4d-4c75-a777-636040b75350-konnectivity-ca\") pod \"konnectivity-agent-gfsnb\" (UID: \"6159f1d7-ac4d-4c75-a777-636040b75350\") " pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:14:40.403232 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403216    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/86301808-2ce5-4813-b527-98b3967b4d15-kubelet-config\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:40.403287 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403237    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-sysctl-d\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.403287 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403252    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-tmp\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.403287 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403269    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-run-openvswitch\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.403287 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403284    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f320b45-0781-45f5-916d-5ac2fe9fa951-ovnkube-script-lib\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.403488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403301    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9stdn\" (UniqueName: \"kubernetes.io/projected/6f320b45-0781-45f5-916d-5ac2fe9fa951-kube-api-access-9stdn\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.403488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403318    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-socket-dir-parent\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.403488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403341    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-host\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.403488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403362    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6m7m\" (UniqueName: \"kubernetes.io/projected/b7a9829f-33e2-4b02-a05a-35f9765d9fbb-kube-api-access-g6m7m\") pod \"node-ca-zsccb\" (UID: \"b7a9829f-33e2-4b02-a05a-35f9765d9fbb\") " pod="openshift-image-registry/node-ca-zsccb"
Jul 24 21:14:40.403488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403378    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-etc-openvswitch\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.403488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403392    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-run-netns\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.403488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403418    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9vj\" (UniqueName: \"kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj\") pod \"network-check-target-9nzl9\" (UID: \"05180a7b-cf89-498d-af4b-6ba6ed49ae0e\") " pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:40.403488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403437    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-socket-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.403488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403452    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7a9829f-33e2-4b02-a05a-35f9765d9fbb-serviceca\") pod \"node-ca-zsccb\" (UID: \"b7a9829f-33e2-4b02-a05a-35f9765d9fbb\") " pod="openshift-image-registry/node-ca-zsccb"
Jul 24 21:14:40.403488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403472    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.403488 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403489    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-cni-bin\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403504    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-hostroot\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403526    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc99v\" (UniqueName: \"kubernetes.io/projected/523501ee-3e85-408e-9dac-a8cee654bc6a-kube-api-access-dc99v\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403549    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-systemd\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403567    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-cni-dir\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403566    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403584    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-var-lib-cni-multus\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403599    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-sys\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403613    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-var-lib-kubelet\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403618    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-registration-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403630    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403748    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-sys-fs\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403765    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-run-netns\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403787    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c62472f-b8c0-4ea0-9d99-5703ddf06036-host-slash\") pod \"iptables-alerter-xlrn4\" (UID: \"2c62472f-b8c0-4ea0-9d99-5703ddf06036\") " pod="openshift-network-operator/iptables-alerter-xlrn4"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403799    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403827    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f320b45-0781-45f5-916d-5ac2fe9fa951-ovnkube-config\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403853    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-cnibin\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.404083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.403953    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-device-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.404855 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.404009    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6460eef-e05a-418d-9c44-36d5d9ac9867-socket-dir\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.406883 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.406793    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal" event={"ID":"6d5eee1a56b131b18e3bafe96ff6f390","Type":"ContainerStarted","Data":"0bdc5e23bef428288bd8d0cfcc9e0e71252be15f12f39c7db814e382eed0eec7"}
Jul 24 21:14:40.408076 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.408049    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-162.ec2.internal" event={"ID":"66217b0fb27a8d11ff5105a5ffa34535","Type":"ContainerStarted","Data":"48ec4233734a7997ad4a26259790963f20771f4cc5081c6473fb987a38fa0c92"}
Jul 24 21:14:40.408959 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.408921    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6159f1d7-ac4d-4c75-a777-636040b75350-agent-certs\") pod \"konnectivity-agent-gfsnb\" (UID: \"6159f1d7-ac4d-4c75-a777-636040b75350\") " pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:14:40.409353 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.409333    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:14:40.409454 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.409358    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:14:40.409454 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.409370    2522 projected.go:196] Error preparing data for projected volume kube-api-access-9l9vj for pod openshift-network-diagnostics/network-check-target-9nzl9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:40.409454 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.409436    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj podName:05180a7b-cf89-498d-af4b-6ba6ed49ae0e nodeName:}" failed. No retries permitted until 2026-07-24 21:14:40.909417662 +0000 UTC m=+2.940812653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9vj" (UniqueName: "kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj") pod "network-check-target-9nzl9" (UID: "05180a7b-cf89-498d-af4b-6ba6ed49ae0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:40.412325 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.412307    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-755fw\" (UniqueName: \"kubernetes.io/projected/2c62472f-b8c0-4ea0-9d99-5703ddf06036-kube-api-access-755fw\") pod \"iptables-alerter-xlrn4\" (UID: \"2c62472f-b8c0-4ea0-9d99-5703ddf06036\") " pod="openshift-network-operator/iptables-alerter-xlrn4"
Jul 24 21:14:40.412751 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.412732    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sglz9\" (UniqueName: \"kubernetes.io/projected/09ad0063-0c05-496f-9696-cec68aa2aa14-kube-api-access-sglz9\") pod \"node-resolver-sw9pw\" (UID: \"09ad0063-0c05-496f-9696-cec68aa2aa14\") " pod="openshift-dns/node-resolver-sw9pw"
Jul 24 21:14:40.414783 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.413742    2522 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6mkrs"
Jul 24 21:14:40.414783 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.414046    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjfz5\" (UniqueName: \"kubernetes.io/projected/a6460eef-e05a-418d-9c44-36d5d9ac9867-kube-api-access-qjfz5\") pod \"aws-ebs-csi-driver-node-2tllz\" (UID: \"a6460eef-e05a-418d-9c44-36d5d9ac9867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.420750 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.420715    2522 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6mkrs"
Jul 24 21:14:40.505086 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505047    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-etc-openvswitch\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.505237 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505094    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-run-netns\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.505237 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505138    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7a9829f-33e2-4b02-a05a-35f9765d9fbb-serviceca\") pod \"node-ca-zsccb\" (UID: \"b7a9829f-33e2-4b02-a05a-35f9765d9fbb\") " pod="openshift-image-registry/node-ca-zsccb"
Jul 24 21:14:40.505237 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505165    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-etc-openvswitch\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.505237 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505166    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.505237 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505231    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-cni-bin\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505238    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-run-netns\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505257    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-hostroot\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505286    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc99v\" (UniqueName: \"kubernetes.io/projected/523501ee-3e85-408e-9dac-a8cee654bc6a-kube-api-access-dc99v\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505311    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-systemd\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505320    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-hostroot\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505288    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-cni-bin\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505373    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-systemd\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505422    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-cni-dir\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505441    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-var-lib-cni-multus\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505487    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-sys\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505510    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-var-lib-kubelet\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.505555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505545    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505567    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-run-netns\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505566    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-var-lib-cni-multus\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505589    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505604    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-sys\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505612    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f320b45-0781-45f5-916d-5ac2fe9fa951-ovnkube-config\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505648    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-cni-dir\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505664    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505678    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-var-lib-kubelet\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505663    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7a9829f-33e2-4b02-a05a-35f9765d9fbb-serviceca\") pod \"node-ca-zsccb\" (UID: \"b7a9829f-33e2-4b02-a05a-35f9765d9fbb\") " pod="openshift-image-registry/node-ca-zsccb"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505738    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-cnibin\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505666    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-cnibin\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505764    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505781    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-run-ovn\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505807    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-run-ovn\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505810    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-conf-dir\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505840    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-etc-kubernetes\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505857    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-conf-dir\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505780    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-run-netns\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505867    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f320b45-0781-45f5-916d-5ac2fe9fa951-ovn-node-metrics-cert\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505893    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-system-cni-dir\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505900    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-etc-kubernetes\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505918    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-kubernetes\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505944    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-systemd-units\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505964    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-system-cni-dir\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505971    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-var-lib-kubelet\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.505999    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-kubelet\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506004    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-systemd-units\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506023    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/523501ee-3e85-408e-9dac-a8cee654bc6a-cni-binary-copy\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506050    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-var-lib-kubelet\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506051    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-var-lib-cni-bin\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506061    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-kubelet\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506026    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-kubernetes\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506086    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm67\" (UniqueName: \"kubernetes.io/projected/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-kube-api-access-kfm67\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506113    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/86301808-2ce5-4813-b527-98b3967b4d15-dbus\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506136    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-lib-modules\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.506916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506160    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f320b45-0781-45f5-916d-5ac2fe9fa951-ovnkube-config\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506165    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-node-log\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506198    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-var-lib-cni-bin\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506214    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/86301808-2ce5-4813-b527-98b3967b4d15-dbus\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506217    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-sysctl-conf\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506242    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-node-log\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506244    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-tuned\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506273    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506299    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-slash\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506329    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-run-systemd\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506354    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-cni-netd\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506377    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-daemon-config\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506400    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-run\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506422    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-slash\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506469    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-run-systemd\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506478    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7a9829f-33e2-4b02-a05a-35f9765d9fbb-host\") pod \"node-ca-zsccb\" (UID: \"b7a9829f-33e2-4b02-a05a-35f9765d9fbb\") " pod="openshift-image-registry/node-ca-zsccb"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506329    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-lib-modules\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506509    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-cni-netd\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.507678 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506559    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/523501ee-3e85-408e-9dac-a8cee654bc6a-cni-binary-copy\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506427    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7a9829f-33e2-4b02-a05a-35f9765d9fbb-host\") pod \"node-ca-zsccb\" (UID: \"b7a9829f-33e2-4b02-a05a-35f9765d9fbb\") " pod="openshift-image-registry/node-ca-zsccb"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506706    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-log-socket\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506723    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506736    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-modprobe-d\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506761    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-sysconfig\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506802    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8z5\" (UniqueName: \"kubernetes.io/projected/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-kube-api-access-fk8z5\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506806    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-log-socket\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506829    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-var-lib-openvswitch\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506854    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-run-k8s-cni-cncf-io\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506862    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-run\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506870    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-sysconfig\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506902    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-sysctl-conf\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506918    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-run-multus-certs\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506957    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-var-lib-openvswitch\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506880    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-run-multus-certs\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.506998    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-modprobe-d\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507002    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-host-run-k8s-cni-cncf-io\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.508317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507024    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-daemon-config\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507110    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507146    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-run-ovn-kubernetes\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507178    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-os-release\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507233    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-host-run-ovn-kubernetes\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507271    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.507238    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507343    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-system-cni-dir\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.507358    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs podName:fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9 nodeName:}" failed. No retries permitted until 2026-07-24 21:14:41.0073369 +0000 UTC m=+3.038731891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs") pod "network-metrics-daemon-krmvj" (UID: "fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507392    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-system-cni-dir\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.507414    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507479    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-os-release\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507248    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-os-release\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.507492    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret podName:86301808-2ce5-4813-b527-98b3967b4d15 nodeName:}" failed. No retries permitted until 2026-07-24 21:14:41.007475963 +0000 UTC m=+3.038870950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret") pod "global-pull-secret-syncer-sxqs9" (UID: "86301808-2ce5-4813-b527-98b3967b4d15") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507387    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-os-release\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507565    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmlg\" (UniqueName: \"kubernetes.io/projected/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-kube-api-access-jcmlg\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.509106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507621    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-cnibin\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507649    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507695    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f320b45-0781-45f5-916d-5ac2fe9fa951-env-overrides\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507722    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/86301808-2ce5-4813-b527-98b3967b4d15-kubelet-config\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507747    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-sysctl-d\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507769    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-tmp\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507793    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-run-openvswitch\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507839    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f320b45-0781-45f5-916d-5ac2fe9fa951-run-openvswitch\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507888    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/86301808-2ce5-4813-b527-98b3967b4d15-kubelet-config\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.507997    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-sysctl-d\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508104    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508134    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f320b45-0781-45f5-916d-5ac2fe9fa951-env-overrides\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508176    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f320b45-0781-45f5-916d-5ac2fe9fa951-ovnkube-script-lib\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508186    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-cnibin\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508194    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9stdn\" (UniqueName: \"kubernetes.io/projected/6f320b45-0781-45f5-916d-5ac2fe9fa951-kube-api-access-9stdn\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508230    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-socket-dir-parent\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508255    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-host\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.509791 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508308    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6m7m\" (UniqueName: \"kubernetes.io/projected/b7a9829f-33e2-4b02-a05a-35f9765d9fbb-kube-api-access-g6m7m\") pod \"node-ca-zsccb\" (UID: \"b7a9829f-33e2-4b02-a05a-35f9765d9fbb\") " pod="openshift-image-registry/node-ca-zsccb"
Jul 24 21:14:40.510415 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508462    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/523501ee-3e85-408e-9dac-a8cee654bc6a-multus-socket-dir-parent\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.510415 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508471    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.510415 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508505    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-host\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.510415 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508738    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f320b45-0781-45f5-916d-5ac2fe9fa951-ovn-node-metrics-cert\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.510415 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.508983    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-etc-tuned\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.510415 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.509015    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f320b45-0781-45f5-916d-5ac2fe9fa951-ovnkube-script-lib\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.512644 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.512616    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-tmp\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.514715 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.514686    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc99v\" (UniqueName: \"kubernetes.io/projected/523501ee-3e85-408e-9dac-a8cee654bc6a-kube-api-access-dc99v\") pod \"multus-r57zx\" (UID: \"523501ee-3e85-408e-9dac-a8cee654bc6a\") " pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.521556 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.521531    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfm67\" (UniqueName: \"kubernetes.io/projected/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-kube-api-access-kfm67\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:40.524989 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.524970    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9stdn\" (UniqueName: \"kubernetes.io/projected/6f320b45-0781-45f5-916d-5ac2fe9fa951-kube-api-access-9stdn\") pod \"ovnkube-node-ntvd4\" (UID: \"6f320b45-0781-45f5-916d-5ac2fe9fa951\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.528957 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.528939    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6m7m\" (UniqueName: \"kubernetes.io/projected/b7a9829f-33e2-4b02-a05a-35f9765d9fbb-kube-api-access-g6m7m\") pod \"node-ca-zsccb\" (UID: \"b7a9829f-33e2-4b02-a05a-35f9765d9fbb\") " pod="openshift-image-registry/node-ca-zsccb"
Jul 24 21:14:40.544230 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.544209    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmlg\" (UniqueName: \"kubernetes.io/projected/82e50870-99f4-4b0c-91ad-dbfca23e5ca7-kube-api-access-jcmlg\") pod \"multus-additional-cni-plugins-xcw5h\" (UID: \"82e50870-99f4-4b0c-91ad-dbfca23e5ca7\") " pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.545950 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.545931    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8z5\" (UniqueName: \"kubernetes.io/projected/9970fe5c-3f27-42a5-adca-5f2fb3b3e935-kube-api-access-fk8z5\") pod \"tuned-xdzf5\" (UID: \"9970fe5c-3f27-42a5-adca-5f2fb3b3e935\") " pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.594052 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.594027    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:14:40.603753 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.603733    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xlrn4"
Jul 24 21:14:40.612903 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.612843    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz"
Jul 24 21:14:40.621592 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.621573    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sw9pw"
Jul 24 21:14:40.629329 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.629309    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zsccb"
Jul 24 21:14:40.638113 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.638094    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xcw5h"
Jul 24 21:14:40.645906 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.645886    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:14:40.653609 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.653590    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdzf5"
Jul 24 21:14:40.659443 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.659425    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r57zx"
Jul 24 21:14:40.911121 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:40.911026    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9vj\" (UniqueName: \"kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj\") pod \"network-check-target-9nzl9\" (UID: \"05180a7b-cf89-498d-af4b-6ba6ed49ae0e\") " pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:40.911277 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.911211    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:14:40.911277 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.911238    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:14:40.911277 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.911263    2522 projected.go:196] Error preparing data for projected volume kube-api-access-9l9vj for pod openshift-network-diagnostics/network-check-target-9nzl9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:40.911400 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:40.911336    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj podName:05180a7b-cf89-498d-af4b-6ba6ed49ae0e nodeName:}" failed. No retries permitted until 2026-07-24 21:14:41.91131085 +0000 UTC m=+3.942705843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9vj" (UniqueName: "kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj") pod "network-check-target-9nzl9" (UID: "05180a7b-cf89-498d-af4b-6ba6ed49ae0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:41.011855 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:41.011809    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:41.011855 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:41.011865    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:41.012088 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:41.011984    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:41.012088 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:41.011993    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:41.012088 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:41.012054    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret podName:86301808-2ce5-4813-b527-98b3967b4d15 nodeName:}" failed. No retries permitted until 2026-07-24 21:14:42.0120343 +0000 UTC m=+4.043429292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret") pod "global-pull-secret-syncer-sxqs9" (UID: "86301808-2ce5-4813-b527-98b3967b4d15") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:41.012088 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:41.012074    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs podName:fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9 nodeName:}" failed. No retries permitted until 2026-07-24 21:14:42.012063946 +0000 UTC m=+4.043458938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs") pod "network-metrics-daemon-krmvj" (UID: "fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:41.402088 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:41.402049    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:41.402751 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:41.402168    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:14:41.426102 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:41.426059    2522 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 21:09:41 +0000 UTC" deadline="2028-02-22 07:00:02.709664516 +0000 UTC"
Jul 24 21:14:41.426102 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:41.426087    2522 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13857h45m21.283582464s"
Jul 24 21:14:41.469906 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:41.469118    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f320b45_0781_45f5_916d_5ac2fe9fa951.slice/crio-5de5c16666ac9b6775db345541207358ee02e48695180d65cd13ce15d0639902 WatchSource:0}: Error finding container 5de5c16666ac9b6775db345541207358ee02e48695180d65cd13ce15d0639902: Status 404 returned error can't find the container with id 5de5c16666ac9b6775db345541207358ee02e48695180d65cd13ce15d0639902
Jul 24 21:14:41.475255 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:41.475235    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6460eef_e05a_418d_9c44_36d5d9ac9867.slice/crio-343fb61a037c87f6ccdaae9ccdd64b956a3f2fc9da3602d84a77f396e6b88d89 WatchSource:0}: Error finding container 343fb61a037c87f6ccdaae9ccdd64b956a3f2fc9da3602d84a77f396e6b88d89: Status 404 returned error can't find the container with id 343fb61a037c87f6ccdaae9ccdd64b956a3f2fc9da3602d84a77f396e6b88d89
Jul 24 21:14:41.478588 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:41.478525    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c62472f_b8c0_4ea0_9d99_5703ddf06036.slice/crio-8434c911e4260ad80c5955045a8739890064a139a001723e3c8bbf524c6f9835 WatchSource:0}: Error finding container 8434c911e4260ad80c5955045a8739890064a139a001723e3c8bbf524c6f9835: Status 404 returned error can't find the container with id 8434c911e4260ad80c5955045a8739890064a139a001723e3c8bbf524c6f9835
Jul 24 21:14:41.480059 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:41.479787    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a9829f_33e2_4b02_a05a_35f9765d9fbb.slice/crio-f1e3bc2654865231a0bd8d2f2f2d759af5cd1a6300afb20fe032a3afc2ce212a WatchSource:0}: Error finding container f1e3bc2654865231a0bd8d2f2f2d759af5cd1a6300afb20fe032a3afc2ce212a: Status 404 returned error can't find the container with id f1e3bc2654865231a0bd8d2f2f2d759af5cd1a6300afb20fe032a3afc2ce212a
Jul 24 21:14:41.481001 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:41.480979    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6159f1d7_ac4d_4c75_a777_636040b75350.slice/crio-e1bbb9c4ef33c2b4e78a6614101b1d18aa95fd42c7a99419cab96b42b9e7a266 WatchSource:0}: Error finding container e1bbb9c4ef33c2b4e78a6614101b1d18aa95fd42c7a99419cab96b42b9e7a266: Status 404 returned error can't find the container with id e1bbb9c4ef33c2b4e78a6614101b1d18aa95fd42c7a99419cab96b42b9e7a266
Jul 24 21:14:41.482976 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:41.482805    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9970fe5c_3f27_42a5_adca_5f2fb3b3e935.slice/crio-f1f94686aa5e868f3e123d25fc76c2e2c5a8dff8d59ffacaeeb8e9cbe7f580d3 WatchSource:0}: Error finding container f1f94686aa5e868f3e123d25fc76c2e2c5a8dff8d59ffacaeeb8e9cbe7f580d3: Status 404 returned error can't find the container with id f1f94686aa5e868f3e123d25fc76c2e2c5a8dff8d59ffacaeeb8e9cbe7f580d3
Jul 24 21:14:41.483758 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:41.483736    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e50870_99f4_4b0c_91ad_dbfca23e5ca7.slice/crio-cd5b906f4227df3905fba0863103413e5dbe49fb6abcb4c676fd3bed92d14611 WatchSource:0}: Error finding container cd5b906f4227df3905fba0863103413e5dbe49fb6abcb4c676fd3bed92d14611: Status 404 returned error can't find the container with id cd5b906f4227df3905fba0863103413e5dbe49fb6abcb4c676fd3bed92d14611
Jul 24 21:14:41.484525 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:14:41.484453    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523501ee_3e85_408e_9dac_a8cee654bc6a.slice/crio-4eca779f46b4c855060864095e766a2f305d69bf0d8ab6d937116e4eb47a2fcc WatchSource:0}: Error finding container 4eca779f46b4c855060864095e766a2f305d69bf0d8ab6d937116e4eb47a2fcc: Status 404 returned error can't find the container with id 4eca779f46b4c855060864095e766a2f305d69bf0d8ab6d937116e4eb47a2fcc
Jul 24 21:14:41.918493 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:41.918301    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9vj\" (UniqueName: \"kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj\") pod \"network-check-target-9nzl9\" (UID: \"05180a7b-cf89-498d-af4b-6ba6ed49ae0e\") " pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:41.918640 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:41.918512    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:14:41.918640 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:41.918539    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:14:41.918640 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:41.918553    2522 projected.go:196] Error preparing data for projected volume kube-api-access-9l9vj for pod openshift-network-diagnostics/network-check-target-9nzl9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:41.918640 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:41.918616    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj podName:05180a7b-cf89-498d-af4b-6ba6ed49ae0e nodeName:}" failed. No retries permitted until 2026-07-24 21:14:43.918594465 +0000 UTC m=+5.949989473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9vj" (UniqueName: "kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj") pod "network-check-target-9nzl9" (UID: "05180a7b-cf89-498d-af4b-6ba6ed49ae0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:42.019218 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.019182    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:42.019385 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.019237    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:42.019460 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:42.019405    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:42.019522 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:42.019500    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret podName:86301808-2ce5-4813-b527-98b3967b4d15 nodeName:}" failed. No retries permitted until 2026-07-24 21:14:44.019480804 +0000 UTC m=+6.050875798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret") pod "global-pull-secret-syncer-sxqs9" (UID: "86301808-2ce5-4813-b527-98b3967b4d15") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:42.019948 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:42.019927    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:42.020042 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:42.019987    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs podName:fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9 nodeName:}" failed. No retries permitted until 2026-07-24 21:14:44.019971154 +0000 UTC m=+6.051366146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs") pod "network-metrics-daemon-krmvj" (UID: "fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:42.401854 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.401812    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:42.402013 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:42.401987    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:14:42.412777 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.403712    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:42.412777 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:42.408981    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:14:42.426662 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.426579    2522 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 21:09:41 +0000 UTC" deadline="2028-01-24 05:56:53.773722114 +0000 UTC"
Jul 24 21:14:42.426662 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.426640    2522 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13160h42m11.34708732s"
Jul 24 21:14:42.448211 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.448146    2522 generic.go:356] "Generic (PLEG): container finished" podID="6d5eee1a56b131b18e3bafe96ff6f390" containerID="0768163210f5ec0d5a03990fafd0d0959c1278bb950351225083a72b9645958c" exitCode=0
Jul 24 21:14:42.448345 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.448244    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal" event={"ID":"6d5eee1a56b131b18e3bafe96ff6f390","Type":"ContainerDied","Data":"0768163210f5ec0d5a03990fafd0d0959c1278bb950351225083a72b9645958c"}
Jul 24 21:14:42.464533 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.464501    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sw9pw" event={"ID":"09ad0063-0c05-496f-9696-cec68aa2aa14","Type":"ContainerStarted","Data":"f7f565eb529f61028abcd4822b21d0e3f85c9a15b0d8594bc68ee753ad607496"}
Jul 24 21:14:42.482425 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.482273    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r57zx" event={"ID":"523501ee-3e85-408e-9dac-a8cee654bc6a","Type":"ContainerStarted","Data":"4eca779f46b4c855060864095e766a2f305d69bf0d8ab6d937116e4eb47a2fcc"}
Jul 24 21:14:42.505864 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.505690    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gfsnb" event={"ID":"6159f1d7-ac4d-4c75-a777-636040b75350","Type":"ContainerStarted","Data":"e1bbb9c4ef33c2b4e78a6614101b1d18aa95fd42c7a99419cab96b42b9e7a266"}
Jul 24 21:14:42.515273 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.515244    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4" event={"ID":"6f320b45-0781-45f5-916d-5ac2fe9fa951","Type":"ContainerStarted","Data":"5de5c16666ac9b6775db345541207358ee02e48695180d65cd13ce15d0639902"}
Jul 24 21:14:42.526102 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.525919    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-162.ec2.internal" event={"ID":"66217b0fb27a8d11ff5105a5ffa34535","Type":"ContainerStarted","Data":"cfb13af266aeb304079a86f96af327ce4e0d069bc30931e49e5295037a4006bc"}
Jul 24 21:14:42.531765 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.531723    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcw5h" event={"ID":"82e50870-99f4-4b0c-91ad-dbfca23e5ca7","Type":"ContainerStarted","Data":"cd5b906f4227df3905fba0863103413e5dbe49fb6abcb4c676fd3bed92d14611"}
Jul 24 21:14:42.541225 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.541163    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xlrn4" event={"ID":"2c62472f-b8c0-4ea0-9d99-5703ddf06036","Type":"ContainerStarted","Data":"8434c911e4260ad80c5955045a8739890064a139a001723e3c8bbf524c6f9835"}
Jul 24 21:14:42.542508 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.542451    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-162.ec2.internal" podStartSLOduration=2.542433847 podStartE2EDuration="2.542433847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:14:42.541729547 +0000 UTC m=+4.573124561" watchObservedRunningTime="2026-07-24 21:14:42.542433847 +0000 UTC m=+4.573828862"
Jul 24 21:14:42.548074 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.548036    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdzf5" event={"ID":"9970fe5c-3f27-42a5-adca-5f2fb3b3e935","Type":"ContainerStarted","Data":"f1f94686aa5e868f3e123d25fc76c2e2c5a8dff8d59ffacaeeb8e9cbe7f580d3"}
Jul 24 21:14:42.550574 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.550544    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zsccb" event={"ID":"b7a9829f-33e2-4b02-a05a-35f9765d9fbb","Type":"ContainerStarted","Data":"f1e3bc2654865231a0bd8d2f2f2d759af5cd1a6300afb20fe032a3afc2ce212a"}
Jul 24 21:14:42.568629 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:42.568562    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz" event={"ID":"a6460eef-e05a-418d-9c44-36d5d9ac9867","Type":"ContainerStarted","Data":"343fb61a037c87f6ccdaae9ccdd64b956a3f2fc9da3602d84a77f396e6b88d89"}
Jul 24 21:14:43.402255 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:43.401745    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:43.402255 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:43.401877    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:14:43.590325 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:43.590288    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal" event={"ID":"6d5eee1a56b131b18e3bafe96ff6f390","Type":"ContainerStarted","Data":"bece5754a1cdecaa29b518f7c35f0072b6f3e9445b9e4baf38de0a5bb8cf6968"}
Jul 24 21:14:43.937309 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:43.936378    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9vj\" (UniqueName: \"kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj\") pod \"network-check-target-9nzl9\" (UID: \"05180a7b-cf89-498d-af4b-6ba6ed49ae0e\") " pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:43.937309 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:43.936548    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:14:43.937309 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:43.936569    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:14:43.937309 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:43.936582    2522 projected.go:196] Error preparing data for projected volume kube-api-access-9l9vj for pod openshift-network-diagnostics/network-check-target-9nzl9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:43.937309 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:43.936646    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj podName:05180a7b-cf89-498d-af4b-6ba6ed49ae0e nodeName:}" failed. No retries permitted until 2026-07-24 21:14:47.936625986 +0000 UTC m=+9.968020977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9vj" (UniqueName: "kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj") pod "network-check-target-9nzl9" (UID: "05180a7b-cf89-498d-af4b-6ba6ed49ae0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:44.037616 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:44.036819    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:44.037616 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:44.036871    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:44.037616 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:44.037027    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:44.037616 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:44.037103    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret podName:86301808-2ce5-4813-b527-98b3967b4d15 nodeName:}" failed. No retries permitted until 2026-07-24 21:14:48.037083912 +0000 UTC m=+10.068478918 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret") pod "global-pull-secret-syncer-sxqs9" (UID: "86301808-2ce5-4813-b527-98b3967b4d15") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:44.037616 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:44.037521    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:44.037616 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:44.037575    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs podName:fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9 nodeName:}" failed. No retries permitted until 2026-07-24 21:14:48.037559178 +0000 UTC m=+10.068954182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs") pod "network-metrics-daemon-krmvj" (UID: "fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:44.403765 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:44.403555    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:44.403765 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:44.403719    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:14:44.404001 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:44.403785    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:44.404001 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:44.403904    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:14:45.401789 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:45.401758    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:45.402316 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:45.401896    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:14:46.403988 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:46.402231    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:46.403988 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:46.402396    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:14:46.403988 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:46.402985    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:46.403988 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:46.403121    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:14:47.401205 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:47.401119    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:47.401374 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:47.401236    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:14:47.970636 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:47.970599    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9vj\" (UniqueName: \"kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj\") pod \"network-check-target-9nzl9\" (UID: \"05180a7b-cf89-498d-af4b-6ba6ed49ae0e\") " pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:47.971155 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:47.970796    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:14:47.971155 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:47.970818    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:14:47.971155 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:47.970828    2522 projected.go:196] Error preparing data for projected volume kube-api-access-9l9vj for pod openshift-network-diagnostics/network-check-target-9nzl9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:47.971155 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:47.970890    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj podName:05180a7b-cf89-498d-af4b-6ba6ed49ae0e nodeName:}" failed. No retries permitted until 2026-07-24 21:14:55.970869178 +0000 UTC m=+18.002264170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9vj" (UniqueName: "kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj") pod "network-check-target-9nzl9" (UID: "05180a7b-cf89-498d-af4b-6ba6ed49ae0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:48.071687 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:48.071647    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:48.071873 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:48.071722    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:48.071943 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:48.071927    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:48.072000 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:48.071982    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret podName:86301808-2ce5-4813-b527-98b3967b4d15 nodeName:}" failed. No retries permitted until 2026-07-24 21:14:56.071967394 +0000 UTC m=+18.103362387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret") pod "global-pull-secret-syncer-sxqs9" (UID: "86301808-2ce5-4813-b527-98b3967b4d15") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:48.072369 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:48.072350    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:48.072444 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:48.072397    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs podName:fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9 nodeName:}" failed. No retries permitted until 2026-07-24 21:14:56.072385028 +0000 UTC m=+18.103780016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs") pod "network-metrics-daemon-krmvj" (UID: "fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:48.403224 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:48.402760    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:48.403224 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:48.402859    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:14:48.403224 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:48.402930    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:48.403224 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:48.403027    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:14:49.401998 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:49.401484    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:49.401998 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:49.401630    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:14:50.402602 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:50.402564    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:50.403036 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:50.402617    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:50.403036 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:50.402756    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:14:50.403036 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:50.402861    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:14:51.401142 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:51.401096    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:51.401337 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:51.401223    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:14:52.402001 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:52.401966    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:52.402001 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:52.402010    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:52.402516 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:52.402128    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:14:52.402516 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:52.402248    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:14:53.401475 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:53.401449    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:53.401651 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:53.401560    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:14:54.401111 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:54.401073    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:54.401541 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:54.401196    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:14:54.401541 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:54.401490    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:54.401662 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:54.401564    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:14:55.401765 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:55.401730    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:55.402222 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:55.401861    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:14:56.038690 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:56.038635    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9vj\" (UniqueName: \"kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj\") pod \"network-check-target-9nzl9\" (UID: \"05180a7b-cf89-498d-af4b-6ba6ed49ae0e\") " pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:56.038868 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:56.038797    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:14:56.038868 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:56.038813    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:14:56.038868 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:56.038822    2522 projected.go:196] Error preparing data for projected volume kube-api-access-9l9vj for pod openshift-network-diagnostics/network-check-target-9nzl9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:56.039025 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:56.038875    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj podName:05180a7b-cf89-498d-af4b-6ba6ed49ae0e nodeName:}" failed. No retries permitted until 2026-07-24 21:15:12.038857468 +0000 UTC m=+34.070252466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9vj" (UniqueName: "kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj") pod "network-check-target-9nzl9" (UID: "05180a7b-cf89-498d-af4b-6ba6ed49ae0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:14:56.139947 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:56.139914    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:56.140148 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:56.140011    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:56.140148 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:56.140054    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:56.140148 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:56.140107    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:56.140148 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:56.140110    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret podName:86301808-2ce5-4813-b527-98b3967b4d15 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:12.140096415 +0000 UTC m=+34.171491401 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret") pod "global-pull-secret-syncer-sxqs9" (UID: "86301808-2ce5-4813-b527-98b3967b4d15") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:14:56.140334 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:56.140164    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs podName:fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:12.140147754 +0000 UTC m=+34.171542746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs") pod "network-metrics-daemon-krmvj" (UID: "fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:14:56.402039 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:56.401955    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:56.402039 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:56.401955    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:56.402494 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:56.402100    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:14:56.402494 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:56.402227    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:14:57.401688 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:57.401656    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:57.401843 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:57.401764    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:14:58.401884 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:58.401851    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:14:58.402313 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:58.401952    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:14:58.402313 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:58.402046    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:14:58.402313 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:58.402168    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:14:59.401269 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:14:59.401237    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:14:59.401444 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:14:59.401359    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:15:00.401285 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:00.401253    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:00.401806 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:00.401378    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:15:00.401806 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:00.401436    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:15:00.401806 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:00.401539    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:15:01.401256 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:01.401222    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:01.401549 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:01.401351    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:15:01.632106 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:01.631664    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r57zx" event={"ID":"523501ee-3e85-408e-9dac-a8cee654bc6a","Type":"ContainerStarted","Data":"b284f42fbbb9d8146ea6a163685e8e0ee47fd8b618374b107771b187c98207ff"}
Jul 24 21:15:01.639320 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:01.639289    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gfsnb" event={"ID":"6159f1d7-ac4d-4c75-a777-636040b75350","Type":"ContainerStarted","Data":"c2369d84ed5c83698518330b70f7fc8cd24a34abc49becf77185f34e49ff5aec"}
Jul 24 21:15:01.644269 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:01.644172    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4" event={"ID":"6f320b45-0781-45f5-916d-5ac2fe9fa951","Type":"ContainerStarted","Data":"31a477eed84cf7de4a31af72e96ad9082ac95366aa411c45acc315bb8402d0ba"}
Jul 24 21:15:01.647143 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:01.647117    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcw5h" event={"ID":"82e50870-99f4-4b0c-91ad-dbfca23e5ca7","Type":"ContainerStarted","Data":"e63a1dedea73cf91d98c80095f3c58579a2ae2b57975e18e6f460c6f371305aa"}
Jul 24 21:15:01.649326 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:01.649297    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdzf5" event={"ID":"9970fe5c-3f27-42a5-adca-5f2fb3b3e935","Type":"ContainerStarted","Data":"d7908bab616369c073a1aaaa4a3f95faaa1e35d6307459a1a0c7663a09f73c27"}
Jul 24 21:15:01.652834 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:01.651314    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r57zx" podStartSLOduration=2.687068821 podStartE2EDuration="22.651303456s" totalImagesPullingTime="19.964234635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:01.64987901 +0000 UTC m=+23.681274042" watchObservedRunningTime="2026-07-24 21:15:01.651303456 +0000 UTC m=+23.682698464"
Jul 24 21:15:01.652834 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:01.651916    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-162.ec2.internal" podStartSLOduration=21.651900783 podStartE2EDuration="21.651900783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:14:43.607329293 +0000 UTC m=+5.638724308" watchObservedRunningTime="2026-07-24 21:15:01.651900783 +0000 UTC m=+23.683295791"
Jul 24 21:15:01.667137 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:01.667092    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gfsnb" podStartSLOduration=2.719143618 podStartE2EDuration="22.667071448s" totalImagesPullingTime="19.94792783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:01.665722017 +0000 UTC m=+23.697117024" watchObservedRunningTime="2026-07-24 21:15:01.667071448 +0000 UTC m=+23.698466456"
Jul 24 21:15:01.704915 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:01.704853    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zsccb" podStartSLOduration=2.750551911 podStartE2EDuration="22.704833148s" totalImagesPullingTime="19.954281237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:01.703258013 +0000 UTC m=+23.734653022" watchObservedRunningTime="2026-07-24 21:15:01.704833148 +0000 UTC m=+23.736228157"
Jul 24 21:15:01.721416 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:01.721374    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xdzf5" podStartSLOduration=2.773640417 podStartE2EDuration="22.721359791s" totalImagesPullingTime="19.947719374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:01.720968082 +0000 UTC m=+23.752363091" watchObservedRunningTime="2026-07-24 21:15:01.721359791 +0000 UTC m=+23.752754798"
Jul 24 21:15:02.296015 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.295780    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:15:02.296461 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.296438    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:15:02.401649 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.401620    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:02.402139 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.401623    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:15:02.402139 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:02.401736    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:15:02.402139 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:02.401808    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:15:02.657260 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.657229    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4" event={"ID":"6f320b45-0781-45f5-916d-5ac2fe9fa951","Type":"ContainerStarted","Data":"8e7d9f3a240eab96796f9a4024d88c005192bc1d8cad7f66c4ef0b3653557360"}
Jul 24 21:15:02.657412 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.657296    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4" event={"ID":"6f320b45-0781-45f5-916d-5ac2fe9fa951","Type":"ContainerStarted","Data":"675a5f9cc052fbcd06f875bfe13df2585d131d17c6a935aadebf134a148567bd"}
Jul 24 21:15:02.657412 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.657336    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4" event={"ID":"6f320b45-0781-45f5-916d-5ac2fe9fa951","Type":"ContainerStarted","Data":"6ef44e4d409b88ad57de80db3bb50b16cc6442d4cc43c75624317e103ed98934"}
Jul 24 21:15:02.657412 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.657374    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4" event={"ID":"6f320b45-0781-45f5-916d-5ac2fe9fa951","Type":"ContainerStarted","Data":"1a4e19ee79b515f119cfad0204867f2663415e4313c6b6c0d3e15fedc6ea3d65"}
Jul 24 21:15:02.657412 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.657408    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4" event={"ID":"6f320b45-0781-45f5-916d-5ac2fe9fa951","Type":"ContainerStarted","Data":"ffdade1238cfa56ce49f4dc662eaa6b518e91a954fb52d0cacc74930dc59417f"}
Jul 24 21:15:02.658600 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.658577    2522 generic.go:356] "Generic (PLEG): container finished" podID="82e50870-99f4-4b0c-91ad-dbfca23e5ca7" containerID="e63a1dedea73cf91d98c80095f3c58579a2ae2b57975e18e6f460c6f371305aa" exitCode=0
Jul 24 21:15:02.658742 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.658654    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcw5h" event={"ID":"82e50870-99f4-4b0c-91ad-dbfca23e5ca7","Type":"ContainerDied","Data":"e63a1dedea73cf91d98c80095f3c58579a2ae2b57975e18e6f460c6f371305aa"}
Jul 24 21:15:02.660097 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.660076    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zsccb" event={"ID":"b7a9829f-33e2-4b02-a05a-35f9765d9fbb","Type":"ContainerStarted","Data":"5ebd66d5aa00337880acd78b66060c4c05810220671c20d324ea85710076020b"}
Jul 24 21:15:02.661718 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.661694    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz" event={"ID":"a6460eef-e05a-418d-9c44-36d5d9ac9867","Type":"ContainerStarted","Data":"65a04bdde3b232737b2a882950dfb65c58c11325cd19d89d70e9a07fce17029f"}
Jul 24 21:15:02.663005 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:02.662983    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sw9pw" event={"ID":"09ad0063-0c05-496f-9696-cec68aa2aa14","Type":"ContainerStarted","Data":"ec2b6d65a19096b3ba9d2c98b1f9e924a8ce127ff9e218d674b245c1b3154e9f"}
Jul 24 21:15:03.066409 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:03.066386    2522 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 21:15:03.363404 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:03.363317    2522 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T21:15:03.066405766Z","UUID":"67ca1552-c86c-45b2-90ab-16f1fdd7490c","Handler":null,"Name":"","Endpoint":""}
Jul 24 21:15:03.365687 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:03.365652    2522 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 21:15:03.365837 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:03.365714    2522 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 21:15:03.401239 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:03.401215    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:03.401367 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:03.401325    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:15:03.667196 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:03.667119    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xlrn4" event={"ID":"2c62472f-b8c0-4ea0-9d99-5703ddf06036","Type":"ContainerStarted","Data":"dc34d520f2a21c0b811dad3503d9dd39509420d7c9437836ac4da99d9a9fdfda"}
Jul 24 21:15:03.669307 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:03.669278    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz" event={"ID":"a6460eef-e05a-418d-9c44-36d5d9ac9867","Type":"ContainerStarted","Data":"250b7320ada7b95ee6d3fe6f085fa4f9d425c2e6fbba9fffc3179ef938be4549"}
Jul 24 21:15:03.669462 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:03.669348    2522 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 21:15:03.682918 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:03.682877    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sw9pw" podStartSLOduration=4.741825517 podStartE2EDuration="24.682858398s" totalImagesPullingTime="19.941032881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:02.698773387 +0000 UTC m=+24.730168395" watchObservedRunningTime="2026-07-24 21:15:03.682858398 +0000 UTC m=+25.714253402"
Jul 24 21:15:04.402038 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:04.401967    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:15:04.402210 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:04.402094    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:15:04.402210 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:04.402118    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:04.402339 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:04.402239    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:15:04.675945 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:04.675906    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4" event={"ID":"6f320b45-0781-45f5-916d-5ac2fe9fa951","Type":"ContainerStarted","Data":"35021fcee999d3476541320c8ea014f0a9f050a6a5e1cab9fb0db142eb59ab90"}
Jul 24 21:15:05.401763 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:05.401730    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:05.402071 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:05.401830    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:15:05.680331 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:05.680043    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz" event={"ID":"a6460eef-e05a-418d-9c44-36d5d9ac9867","Type":"ContainerStarted","Data":"ae242826331f9adf92b611574a7567b42c8f059359c492f6ac65a7e0a147c761"}
Jul 24 21:15:05.698037 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:05.697990    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xlrn4" podStartSLOduration=6.747199333 podStartE2EDuration="26.697974678s" totalImagesPullingTime="19.950775345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:03.683199663 +0000 UTC m=+25.714594671" watchObservedRunningTime="2026-07-24 21:15:05.697974678 +0000 UTC m=+27.729369715"
Jul 24 21:15:05.698311 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:05.698282    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tllz" podStartSLOduration=3.980768947 podStartE2EDuration="26.698271217s" totalImagesPullingTime="22.71750227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:05.697580462 +0000 UTC m=+27.728975471" watchObservedRunningTime="2026-07-24 21:15:05.698271217 +0000 UTC m=+27.729666226"
Jul 24 21:15:06.402056 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:06.402024    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:15:06.402256 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:06.402217    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:15:06.402256 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:06.402255    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:06.402450 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:06.402340    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:15:07.401420 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:07.401257    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:07.402069 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:07.401484    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:15:07.685235 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:07.685164    2522 generic.go:356] "Generic (PLEG): container finished" podID="82e50870-99f4-4b0c-91ad-dbfca23e5ca7" containerID="0f47f38e918802128fe892650b1f8cf0e7bbca8aa74f8eb9307ff4e9ad9484fe" exitCode=0
Jul 24 21:15:07.685404 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:07.685248    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcw5h" event={"ID":"82e50870-99f4-4b0c-91ad-dbfca23e5ca7","Type":"ContainerDied","Data":"0f47f38e918802128fe892650b1f8cf0e7bbca8aa74f8eb9307ff4e9ad9484fe"}
Jul 24 21:15:07.689215 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:07.689191    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4" event={"ID":"6f320b45-0781-45f5-916d-5ac2fe9fa951","Type":"ContainerStarted","Data":"36873154dc16b0a31b0b485e69aa80a50764c0144d05a3ebd03e9cc95b14aaf5"}
Jul 24 21:15:07.690340 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:07.689489    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:15:07.690340 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:07.689558    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:15:07.705652 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:07.705632    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:15:07.705931 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:07.705914    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:15:07.738792 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:07.738752    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4" podStartSLOduration=8.693967637 podStartE2EDuration="28.738736469s" totalImagesPullingTime="20.044768832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:07.737998753 +0000 UTC m=+29.769393760" watchObservedRunningTime="2026-07-24 21:15:07.738736469 +0000 UTC m=+29.770131479"
Jul 24 21:15:08.404465 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:08.404434    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:15:08.405015 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:08.404617    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:15:08.405015 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:08.404876    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:08.405015 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:08.405001    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:15:08.685367 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:08.685330    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9nzl9"]
Jul 24 21:15:08.685465 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:08.685447    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:08.685549 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:08.685530    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:15:08.689604 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:08.689578    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sxqs9"]
Jul 24 21:15:08.691058 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:08.691034    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-krmvj"]
Jul 24 21:15:08.693913 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:08.693876    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcw5h" event={"ID":"82e50870-99f4-4b0c-91ad-dbfca23e5ca7","Type":"ContainerStarted","Data":"08c30cf8ef027bfa6779b26eecc1c59a150a4e3b11758e2ea9c26ffc00d3d2d7"}
Jul 24 21:15:08.693913 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:08.693906    2522 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 21:15:08.694248 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:08.693961    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:08.694248 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:08.694061    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:15:08.694365 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:08.694358    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:15:08.694446 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:08.694425    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:15:09.096543 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:09.096518    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:15:09.096723 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:09.096624    2522 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 21:15:09.097196 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:09.097166    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gfsnb"
Jul 24 21:15:09.698278 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:09.698250    2522 generic.go:356] "Generic (PLEG): container finished" podID="82e50870-99f4-4b0c-91ad-dbfca23e5ca7" containerID="08c30cf8ef027bfa6779b26eecc1c59a150a4e3b11758e2ea9c26ffc00d3d2d7" exitCode=0
Jul 24 21:15:09.698657 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:09.698338    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcw5h" event={"ID":"82e50870-99f4-4b0c-91ad-dbfca23e5ca7","Type":"ContainerDied","Data":"08c30cf8ef027bfa6779b26eecc1c59a150a4e3b11758e2ea9c26ffc00d3d2d7"}
Jul 24 21:15:09.698657 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:09.698453    2522 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 21:15:10.050876 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:10.050811    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:15:10.401724 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:10.401648    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:10.401977 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:10.401660    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:15:10.401977 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:10.401772    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:15:10.401977 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:10.401875    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:10.401977 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:10.401869    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:15:10.402210 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:10.401988    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:15:11.704894 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:11.704717    2522 generic.go:356] "Generic (PLEG): container finished" podID="82e50870-99f4-4b0c-91ad-dbfca23e5ca7" containerID="d5afc3df2b5b4c28549e10cca4f70f2482ff58d7deeee74a8faeea8659fef8ce" exitCode=0
Jul 24 21:15:11.704894 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:11.704778    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcw5h" event={"ID":"82e50870-99f4-4b0c-91ad-dbfca23e5ca7","Type":"ContainerDied","Data":"d5afc3df2b5b4c28549e10cca4f70f2482ff58d7deeee74a8faeea8659fef8ce"}
Jul 24 21:15:12.067862 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:12.067798    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9vj\" (UniqueName: \"kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj\") pod \"network-check-target-9nzl9\" (UID: \"05180a7b-cf89-498d-af4b-6ba6ed49ae0e\") " pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:12.067960 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:12.067944    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:12.067960 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:12.067959    2522 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:12.068028 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:12.067968    2522 projected.go:196] Error preparing data for projected volume kube-api-access-9l9vj for pod openshift-network-diagnostics/network-check-target-9nzl9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:12.068028 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:12.068017    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj podName:05180a7b-cf89-498d-af4b-6ba6ed49ae0e nodeName:}" failed. No retries permitted until 2026-07-24 21:15:44.068000514 +0000 UTC m=+66.099395500 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9l9vj" (UniqueName: "kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj") pod "network-check-target-9nzl9" (UID: "05180a7b-cf89-498d-af4b-6ba6ed49ae0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:12.168528 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:12.168503    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:15:12.168632 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:12.168534    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:12.168715 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:12.168635    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:12.168715 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:12.168635    2522 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:12.168715 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:12.168708    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret podName:86301808-2ce5-4813-b527-98b3967b4d15 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:44.168691798 +0000 UTC m=+66.200086808 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret") pod "global-pull-secret-syncer-sxqs9" (UID: "86301808-2ce5-4813-b527-98b3967b4d15") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:12.168826 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:12.168726    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs podName:fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:44.16871652 +0000 UTC m=+66.200111521 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs") pod "network-metrics-daemon-krmvj" (UID: "fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:12.401477 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:12.401404    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:12.401477 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:12.401423    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:15:12.401664 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:12.401524    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9nzl9" podUID="05180a7b-cf89-498d-af4b-6ba6ed49ae0e"
Jul 24 21:15:12.401664 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:12.401595    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krmvj" podUID="fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9"
Jul 24 21:15:12.401664 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:12.401634    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:12.401797 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:12.401750    2522 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxqs9" podUID="86301808-2ce5-4813-b527-98b3967b4d15"
Jul 24 21:15:14.177280 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.177249    2522 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-162.ec2.internal" event="NodeReady"
Jul 24 21:15:14.177948 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.177395    2522 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 21:15:14.212287 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.212253    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67c49bbfb6-9t74k"]
Jul 24 21:15:14.216352 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.216329    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.218606 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.218585    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 21:15:14.218722 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.218594    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 21:15:14.218947 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.218932    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 21:15:14.219069 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.219043    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rzr52\""
Jul 24 21:15:14.224855 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.224458    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 21:15:14.227985 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.227964    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-krwsr"]
Jul 24 21:15:14.231436 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.231419    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67c49bbfb6-9t74k"]
Jul 24 21:15:14.231528 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.231495    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.234177 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.234147    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 21:15:14.234294 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.234190    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wbvbs\""
Jul 24 21:15:14.234294 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.234156    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 21:15:14.234451 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.234379    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j7h2p"]
Jul 24 21:15:14.237617 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.237597    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j7h2p"
Jul 24 21:15:14.239980 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.239961    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-krwsr"]
Jul 24 21:15:14.240619 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.240600    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 21:15:14.240619 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.240613    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ng82g\""
Jul 24 21:15:14.241197 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.241015    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 21:15:14.242564 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.242544    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 21:15:14.248809 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.248387    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j7h2p"]
Jul 24 21:15:14.386999 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.386970    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/104d73c0-ff20-47b8-9774-6b3eaa2708ae-ca-trust-extracted\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.387160 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387014    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-installation-pull-secrets\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.387160 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387093    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-trusted-ca\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.387160 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387122    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b257t\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-kube-api-access-b257t\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.387160 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387146    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.387337 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387171    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ff7824c-d44f-48f1-af97-518f5b3b5d9b-cert\") pod \"ingress-canary-j7h2p\" (UID: \"7ff7824c-d44f-48f1-af97-518f5b3b5d9b\") " pod="openshift-ingress-canary/ingress-canary-j7h2p"
Jul 24 21:15:14.387337 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387200    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-bound-sa-token\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.387337 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387260    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e21e54d-fb5e-4a9a-9f64-908449a5e755-config-volume\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.387337 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387326    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-image-registry-private-configuration\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.387452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387356    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.387452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387375    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbs48\" (UniqueName: \"kubernetes.io/projected/7ff7824c-d44f-48f1-af97-518f5b3b5d9b-kube-api-access-mbs48\") pod \"ingress-canary-j7h2p\" (UID: \"7ff7824c-d44f-48f1-af97-518f5b3b5d9b\") " pod="openshift-ingress-canary/ingress-canary-j7h2p"
Jul 24 21:15:14.387452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387392    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-certificates\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.387452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387412    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e21e54d-fb5e-4a9a-9f64-908449a5e755-tmp-dir\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.387608 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.387462    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbvl\" (UniqueName: \"kubernetes.io/projected/1e21e54d-fb5e-4a9a-9f64-908449a5e755-kube-api-access-8lbvl\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.401204 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.401182    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:14.401337 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.401183    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:14.401401 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.401183    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:15:14.403955 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.403932    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 21:15:14.404054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.403984    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 21:15:14.404382 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.404365    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qnp5w\""
Jul 24 21:15:14.404486 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.404395    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 21:15:14.404545 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.404487    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 21:15:14.404722 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.404701    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zlbx8\""
Jul 24 21:15:14.488175 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488115    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbvl\" (UniqueName: \"kubernetes.io/projected/1e21e54d-fb5e-4a9a-9f64-908449a5e755-kube-api-access-8lbvl\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.488175 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488148    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/104d73c0-ff20-47b8-9774-6b3eaa2708ae-ca-trust-extracted\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.488322 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488180    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-installation-pull-secrets\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.488466 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488448    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-trusted-ca\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.488527 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488486    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b257t\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-kube-api-access-b257t\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.488527 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488509    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.488633 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488532    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ff7824c-d44f-48f1-af97-518f5b3b5d9b-cert\") pod \"ingress-canary-j7h2p\" (UID: \"7ff7824c-d44f-48f1-af97-518f5b3b5d9b\") " pod="openshift-ingress-canary/ingress-canary-j7h2p"
Jul 24 21:15:14.488633 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488531    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/104d73c0-ff20-47b8-9774-6b3eaa2708ae-ca-trust-extracted\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.488633 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488572    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-bound-sa-token\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.488633 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488607    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e21e54d-fb5e-4a9a-9f64-908449a5e755-config-volume\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.488858 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488660    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-image-registry-private-configuration\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.488858 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488710    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.488858 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488733    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbs48\" (UniqueName: \"kubernetes.io/projected/7ff7824c-d44f-48f1-af97-518f5b3b5d9b-kube-api-access-mbs48\") pod \"ingress-canary-j7h2p\" (UID: \"7ff7824c-d44f-48f1-af97-518f5b3b5d9b\") " pod="openshift-ingress-canary/ingress-canary-j7h2p"
Jul 24 21:15:14.488858 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488765    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-certificates\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.488858 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.488788    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e21e54d-fb5e-4a9a-9f64-908449a5e755-tmp-dir\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.488858 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:14.488810    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:14.489123 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:14.488868    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls podName:1e21e54d-fb5e-4a9a-9f64-908449a5e755 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:14.988853204 +0000 UTC m=+37.020248191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls") pod "dns-default-krwsr" (UID: "1e21e54d-fb5e-4a9a-9f64-908449a5e755") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:14.489185 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.489146    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e21e54d-fb5e-4a9a-9f64-908449a5e755-tmp-dir\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.489259 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:14.489238    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:15:14.489259 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:14.489259    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67c49bbfb6-9t74k: secret "image-registry-tls" not found
Jul 24 21:15:14.489540 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:14.489318    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls podName:104d73c0-ff20-47b8-9774-6b3eaa2708ae nodeName:}" failed. No retries permitted until 2026-07-24 21:15:14.989301111 +0000 UTC m=+37.020696116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls") pod "image-registry-67c49bbfb6-9t74k" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae") : secret "image-registry-tls" not found
Jul 24 21:15:14.489938 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.489907    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-certificates\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.490081 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.490060    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e21e54d-fb5e-4a9a-9f64-908449a5e755-config-volume\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.490320 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.490291    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-trusted-ca\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.493718 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.493699    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ff7824c-d44f-48f1-af97-518f5b3b5d9b-cert\") pod \"ingress-canary-j7h2p\" (UID: \"7ff7824c-d44f-48f1-af97-518f5b3b5d9b\") " pod="openshift-ingress-canary/ingress-canary-j7h2p"
Jul 24 21:15:14.494035 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.494016    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-image-registry-private-configuration\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.498694 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.498065    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-installation-pull-secrets\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.498694 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.498626    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b257t\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-kube-api-access-b257t\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.498878 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.498840    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbvl\" (UniqueName: \"kubernetes.io/projected/1e21e54d-fb5e-4a9a-9f64-908449a5e755-kube-api-access-8lbvl\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.499364 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.499344    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-bound-sa-token\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.499453 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.499430    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbs48\" (UniqueName: \"kubernetes.io/projected/7ff7824c-d44f-48f1-af97-518f5b3b5d9b-kube-api-access-mbs48\") pod \"ingress-canary-j7h2p\" (UID: \"7ff7824c-d44f-48f1-af97-518f5b3b5d9b\") " pod="openshift-ingress-canary/ingress-canary-j7h2p"
Jul 24 21:15:14.551542 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.551516    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j7h2p"
Jul 24 21:15:14.993176 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.993140    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:14.993338 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:14.993245    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:14.993338 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:14.993306    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:15:14.993338 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:14.993325    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67c49bbfb6-9t74k: secret "image-registry-tls" not found
Jul 24 21:15:14.993499 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:14.993382    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:14.993499 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:14.993390    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls podName:104d73c0-ff20-47b8-9774-6b3eaa2708ae nodeName:}" failed. No retries permitted until 2026-07-24 21:15:15.993367482 +0000 UTC m=+38.024762486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls") pod "image-registry-67c49bbfb6-9t74k" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae") : secret "image-registry-tls" not found
Jul 24 21:15:14.993499 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:14.993447    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls podName:1e21e54d-fb5e-4a9a-9f64-908449a5e755 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:15.993429927 +0000 UTC m=+38.024824920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls") pod "dns-default-krwsr" (UID: "1e21e54d-fb5e-4a9a-9f64-908449a5e755") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:15.072181 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:15.072137    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:15.156453 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:15.156420    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:15.534409 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:15.534357    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:15.635207 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:15.635173    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:15.853009 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:15.852985    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j7h2p"]
Jul 24 21:15:15.861590 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:15:15.861566    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff7824c_d44f_48f1_af97_518f5b3b5d9b.slice/crio-cd4f29c251be821688d2ffcb1c832a3cd1f29cec4e766332f96884bd8d4e34a3 WatchSource:0}: Error finding container cd4f29c251be821688d2ffcb1c832a3cd1f29cec4e766332f96884bd8d4e34a3: Status 404 returned error can't find the container with id cd4f29c251be821688d2ffcb1c832a3cd1f29cec4e766332f96884bd8d4e34a3
Jul 24 21:15:15.935272 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:15.935241    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:16.006309 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:16.006284    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:16.006395 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:16.006333    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:16.006442 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:16.006424    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:15:16.006442 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:16.006425    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:16.006442 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:16.006436    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67c49bbfb6-9t74k: secret "image-registry-tls" not found
Jul 24 21:15:16.006539 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:16.006489    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls podName:104d73c0-ff20-47b8-9774-6b3eaa2708ae nodeName:}" failed. No retries permitted until 2026-07-24 21:15:18.006475439 +0000 UTC m=+40.037870425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls") pod "image-registry-67c49bbfb6-9t74k" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae") : secret "image-registry-tls" not found
Jul 24 21:15:16.006539 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:16.006501    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls podName:1e21e54d-fb5e-4a9a-9f64-908449a5e755 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:18.006494927 +0000 UTC m=+40.037889913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls") pod "dns-default-krwsr" (UID: "1e21e54d-fb5e-4a9a-9f64-908449a5e755") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:16.035594 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:16.035536    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:16.717142 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:16.717103    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j7h2p" event={"ID":"7ff7824c-d44f-48f1-af97-518f5b3b5d9b","Type":"ContainerStarted","Data":"cd4f29c251be821688d2ffcb1c832a3cd1f29cec4e766332f96884bd8d4e34a3"}
Jul 24 21:15:16.720368 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:16.720338    2522 generic.go:356] "Generic (PLEG): container finished" podID="82e50870-99f4-4b0c-91ad-dbfca23e5ca7" containerID="f0e4a046acf00096f2af61fe6b98b78aa2f97702d974903c8a3669c308e1a07c" exitCode=0
Jul 24 21:15:16.720482 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:16.720383    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcw5h" event={"ID":"82e50870-99f4-4b0c-91ad-dbfca23e5ca7","Type":"ContainerDied","Data":"f0e4a046acf00096f2af61fe6b98b78aa2f97702d974903c8a3669c308e1a07c"}
Jul 24 21:15:17.719416 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:17.719385    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:17.726830 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:17.726803    2522 generic.go:356] "Generic (PLEG): container finished" podID="82e50870-99f4-4b0c-91ad-dbfca23e5ca7" containerID="8bbcf39ae5bc35786fd34423e41d09915cd9198d012e92e2629980845cc3ffbc" exitCode=0
Jul 24 21:15:17.726951 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:17.726848    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcw5h" event={"ID":"82e50870-99f4-4b0c-91ad-dbfca23e5ca7","Type":"ContainerDied","Data":"8bbcf39ae5bc35786fd34423e41d09915cd9198d012e92e2629980845cc3ffbc"}
Jul 24 21:15:17.820166 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:17.820136    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:18.021031 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:18.020955    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:18.021031 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:18.021021    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:18.021233 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:18.021129    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:15:18.021233 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:18.021129    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:18.021233 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:18.021213    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls podName:1e21e54d-fb5e-4a9a-9f64-908449a5e755 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:22.021191723 +0000 UTC m=+44.052586728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls") pod "dns-default-krwsr" (UID: "1e21e54d-fb5e-4a9a-9f64-908449a5e755") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:18.021362 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:18.021140    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67c49bbfb6-9t74k: secret "image-registry-tls" not found
Jul 24 21:15:18.021362 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:18.021298    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls podName:104d73c0-ff20-47b8-9774-6b3eaa2708ae nodeName:}" failed. No retries permitted until 2026-07-24 21:15:22.021280558 +0000 UTC m=+44.052675545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls") pod "image-registry-67c49bbfb6-9t74k" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae") : secret "image-registry-tls" not found
Jul 24 21:15:18.731017 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:18.730802    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j7h2p" event={"ID":"7ff7824c-d44f-48f1-af97-518f5b3b5d9b","Type":"ContainerStarted","Data":"eaafa040d441e3321911bddd1409a4b6d9565a5bce89a705d0d25594ed0501ba"}
Jul 24 21:15:18.734841 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:18.734816    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcw5h" event={"ID":"82e50870-99f4-4b0c-91ad-dbfca23e5ca7","Type":"ContainerStarted","Data":"10643fd8381866a773782e390e83c42ed158d7fa7e52731b8a9c189a9cc3f630"}
Jul 24 21:15:18.746467 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:18.746417    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j7h2p" podStartSLOduration=2.1041488839999998 podStartE2EDuration="4.746403642s" totalImagesPullingTime="2.642254758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:18.746220909 +0000 UTC m=+40.777615921" watchObservedRunningTime="2026-07-24 21:15:18.746403642 +0000 UTC m=+40.777798650"
Jul 24 21:15:18.770717 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:18.770649    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xcw5h" podStartSLOduration=9.785587261 podStartE2EDuration="39.770627523s" totalImagesPullingTime="29.985040262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:18.7693315 +0000 UTC m=+40.800726534" watchObservedRunningTime="2026-07-24 21:15:18.770627523 +0000 UTC m=+40.802022532"
Jul 24 21:15:18.894646 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:18.894568    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:18.952745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:18.952709    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:22.048411 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:22.048372    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:22.049017 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:22.048426    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:22.049017 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:22.048521    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:22.049017 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:22.048598    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls podName:1e21e54d-fb5e-4a9a-9f64-908449a5e755 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:30.048578264 +0000 UTC m=+52.079973270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls") pod "dns-default-krwsr" (UID: "1e21e54d-fb5e-4a9a-9f64-908449a5e755") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:22.049017 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:22.048531    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:15:22.049017 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:22.048624    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67c49bbfb6-9t74k: secret "image-registry-tls" not found
Jul 24 21:15:22.049017 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:22.048697    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls podName:104d73c0-ff20-47b8-9774-6b3eaa2708ae nodeName:}" failed. No retries permitted until 2026-07-24 21:15:30.048661489 +0000 UTC m=+52.080056476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls") pod "image-registry-67c49bbfb6-9t74k" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae") : secret "image-registry-tls" not found
Jul 24 21:15:23.332314 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:23.332268    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:23.456085 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:23.456048    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:23.556978 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:23.556925    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:30.105924 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:30.105891    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:30.106436 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:30.105953    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:30.106436 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:30.106037    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:30.106436 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:30.106054    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:15:30.106436 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:30.106079    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67c49bbfb6-9t74k: secret "image-registry-tls" not found
Jul 24 21:15:30.106436 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:30.106086    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls podName:1e21e54d-fb5e-4a9a-9f64-908449a5e755 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:46.106073064 +0000 UTC m=+68.137468051 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls") pod "dns-default-krwsr" (UID: "1e21e54d-fb5e-4a9a-9f64-908449a5e755") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:30.106436 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:30.106146    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls podName:104d73c0-ff20-47b8-9774-6b3eaa2708ae nodeName:}" failed. No retries permitted until 2026-07-24 21:15:46.106130235 +0000 UTC m=+68.137525221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls") pod "image-registry-67c49bbfb6-9t74k" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae") : secret "image-registry-tls" not found
Jul 24 21:15:36.402531 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:36.402484    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:36.462254 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:36.462210    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:36.531467 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:36.531438    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:36.636644 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:36.636611    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:36.737282 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:36.737251    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:37.465461 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:37.465424    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:37.528046 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:37.528003    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:39.392198 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:39.392164    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:40.723159 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:40.723126    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntvd4"
Jul 24 21:15:40.878249 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:40.878200    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:40.976956 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:40.976874    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:44.103031 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.102989    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9vj\" (UniqueName: \"kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj\") pod \"network-check-target-9nzl9\" (UID: \"05180a7b-cf89-498d-af4b-6ba6ed49ae0e\") " pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:44.105434 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.105412    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 21:15:44.115804 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.115786    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 21:15:44.128375 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.128345    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9vj\" (UniqueName: \"kubernetes.io/projected/05180a7b-cf89-498d-af4b-6ba6ed49ae0e-kube-api-access-9l9vj\") pod \"network-check-target-9nzl9\" (UID: \"05180a7b-cf89-498d-af4b-6ba6ed49ae0e\") " pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:44.203377 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.203343    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:15:44.203377 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.203379    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:44.205906 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.205883    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 21:15:44.205971 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.205907    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 21:15:44.214247 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:44.214215    2522 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 21:15:44.214342 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:44.214298    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs podName:fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:48.214281938 +0000 UTC m=+130.245676949 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs") pod "network-metrics-daemon-krmvj" (UID: "fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9") : secret "metrics-daemon-secret" not found
Jul 24 21:15:44.216648 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.216623    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86301808-2ce5-4813-b527-98b3967b4d15-original-pull-secret\") pod \"global-pull-secret-syncer-sxqs9\" (UID: \"86301808-2ce5-4813-b527-98b3967b4d15\") " pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:44.412329 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.412272    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxqs9"
Jul 24 21:15:44.423079 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.423052    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qnp5w\""
Jul 24 21:15:44.431218 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.431192    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:44.546361 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.546332    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sxqs9"]
Jul 24 21:15:44.549193 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:15:44.549168    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86301808_2ce5_4813_b527_98b3967b4d15.slice/crio-7b8b6b855305d8f66662330471e1cc996e923fd9928ed25e0bfed294c0ca96dd WatchSource:0}: Error finding container 7b8b6b855305d8f66662330471e1cc996e923fd9928ed25e0bfed294c0ca96dd: Status 404 returned error can't find the container with id 7b8b6b855305d8f66662330471e1cc996e923fd9928ed25e0bfed294c0ca96dd
Jul 24 21:15:44.559916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.559894    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9nzl9"]
Jul 24 21:15:44.562286 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:15:44.562261    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05180a7b_cf89_498d_af4b_6ba6ed49ae0e.slice/crio-2a202b2baa2072ad36384cb15a6dce34b3f2bc5e19a3be6e8278b37341039cde WatchSource:0}: Error finding container 2a202b2baa2072ad36384cb15a6dce34b3f2bc5e19a3be6e8278b37341039cde: Status 404 returned error can't find the container with id 2a202b2baa2072ad36384cb15a6dce34b3f2bc5e19a3be6e8278b37341039cde
Jul 24 21:15:44.800489 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.800454    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sxqs9" event={"ID":"86301808-2ce5-4813-b527-98b3967b4d15","Type":"ContainerStarted","Data":"7b8b6b855305d8f66662330471e1cc996e923fd9928ed25e0bfed294c0ca96dd"}
Jul 24 21:15:44.801502 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:44.801482    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9nzl9" event={"ID":"05180a7b-cf89-498d-af4b-6ba6ed49ae0e","Type":"ContainerStarted","Data":"2a202b2baa2072ad36384cb15a6dce34b3f2bc5e19a3be6e8278b37341039cde"}
Jul 24 21:15:45.308508 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:45.308447    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:45.381524 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:45.381490    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:46.117540 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:46.117504    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:15:46.117814 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:46.117569    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:15:46.117814 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:46.117680    2522 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:46.117814 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:46.117749    2522 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:15:46.117814 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:46.117763    2522 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67c49bbfb6-9t74k: secret "image-registry-tls" not found
Jul 24 21:15:46.117814 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:46.117764    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls podName:1e21e54d-fb5e-4a9a-9f64-908449a5e755 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:18.117742739 +0000 UTC m=+100.149137746 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls") pod "dns-default-krwsr" (UID: "1e21e54d-fb5e-4a9a-9f64-908449a5e755") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:46.118067 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:46.117833    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls podName:104d73c0-ff20-47b8-9774-6b3eaa2708ae nodeName:}" failed. No retries permitted until 2026-07-24 21:16:18.117816272 +0000 UTC m=+100.149211261 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls") pod "image-registry-67c49bbfb6-9t74k" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae") : secret "image-registry-tls" not found
Jul 24 21:15:47.014988 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.014943    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"]
Jul 24 21:15:47.050663 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.050637    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-bsshq"]
Jul 24 21:15:47.050829 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.050793    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:47.056957 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.056910    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 24 21:15:47.056957 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.056919    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x762r\""
Jul 24 21:15:47.056957 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.056924    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 21:15:47.057166 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.056988    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 21:15:47.057418 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.057260    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 24 21:15:47.066151 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.066129    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"]
Jul 24 21:15:47.066291 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.066268    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bsshq"
Jul 24 21:15:47.068475 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.068446    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dffds\""
Jul 24 21:15:47.084198 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.084177    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"]
Jul 24 21:15:47.084298 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.084218    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"]
Jul 24 21:15:47.084298 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.084244    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-bsshq"]
Jul 24 21:15:47.084298 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.084293    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"
Jul 24 21:15:47.086490 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.086470    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:47.086719 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.086705    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 24 21:15:47.088906 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.088886    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:15:47.089222 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.089202    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 24 21:15:47.089306 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.089242    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m524f\""
Jul 24 21:15:47.089306 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.089286    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 24 21:15:47.125510 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.125483    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rht2z\" (UniqueName: \"kubernetes.io/projected/27b45efa-aab7-4a1f-a6fb-3a701fdb7d89-kube-api-access-rht2z\") pod \"network-check-source-7d88cb55d8-bsshq\" (UID: \"27b45efa-aab7-4a1f-a6fb-3a701fdb7d89\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bsshq"
Jul 24 21:15:47.125639 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.125523    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:47.125639 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.125550    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96f53b85-bcf8-4a28-9a05-5738e0e9e873-config\") pod \"service-ca-operator-5d6c75f84f-llmz9\" (UID: \"96f53b85-bcf8-4a28-9a05-5738e0e9e873\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"
Jul 24 21:15:47.125800 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.125732    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47g9l\" (UniqueName: \"kubernetes.io/projected/96f53b85-bcf8-4a28-9a05-5738e0e9e873-kube-api-access-47g9l\") pod \"service-ca-operator-5d6c75f84f-llmz9\" (UID: \"96f53b85-bcf8-4a28-9a05-5738e0e9e873\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"
Jul 24 21:15:47.125800 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.125782    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lkfs\" (UniqueName: \"kubernetes.io/projected/689dd5e0-863e-4f54-af4b-3c268bfe6351-kube-api-access-9lkfs\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:47.125905 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.125815    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/689dd5e0-863e-4f54-af4b-3c268bfe6351-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:47.125905 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.125856    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96f53b85-bcf8-4a28-9a05-5738e0e9e873-serving-cert\") pod \"service-ca-operator-5d6c75f84f-llmz9\" (UID: \"96f53b85-bcf8-4a28-9a05-5738e0e9e873\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"
Jul 24 21:15:47.175738 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.175648    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:47.213562 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.213529    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-749656c945-wnf2b"]
Jul 24 21:15:47.226528 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.226500    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47g9l\" (UniqueName: \"kubernetes.io/projected/96f53b85-bcf8-4a28-9a05-5738e0e9e873-kube-api-access-47g9l\") pod \"service-ca-operator-5d6c75f84f-llmz9\" (UID: \"96f53b85-bcf8-4a28-9a05-5738e0e9e873\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"
Jul 24 21:15:47.226629 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.226549    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lkfs\" (UniqueName: \"kubernetes.io/projected/689dd5e0-863e-4f54-af4b-3c268bfe6351-kube-api-access-9lkfs\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:47.226629 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.226593    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/689dd5e0-863e-4f54-af4b-3c268bfe6351-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:47.226754 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.226632    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96f53b85-bcf8-4a28-9a05-5738e0e9e873-serving-cert\") pod \"service-ca-operator-5d6c75f84f-llmz9\" (UID: \"96f53b85-bcf8-4a28-9a05-5738e0e9e873\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"
Jul 24 21:15:47.226754 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.226705    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rht2z\" (UniqueName: \"kubernetes.io/projected/27b45efa-aab7-4a1f-a6fb-3a701fdb7d89-kube-api-access-rht2z\") pod \"network-check-source-7d88cb55d8-bsshq\" (UID: \"27b45efa-aab7-4a1f-a6fb-3a701fdb7d89\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bsshq"
Jul 24 21:15:47.226754 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.226734    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:47.226876 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.226762    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96f53b85-bcf8-4a28-9a05-5738e0e9e873-config\") pod \"service-ca-operator-5d6c75f84f-llmz9\" (UID: \"96f53b85-bcf8-4a28-9a05-5738e0e9e873\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"
Jul 24 21:15:47.227295 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.227274    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96f53b85-bcf8-4a28-9a05-5738e0e9e873-config\") pod \"service-ca-operator-5d6c75f84f-llmz9\" (UID: \"96f53b85-bcf8-4a28-9a05-5738e0e9e873\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"
Jul 24 21:15:47.227384 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:47.227370    2522 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:15:47.227464 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:47.227453    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls podName:689dd5e0-863e-4f54-af4b-3c268bfe6351 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:47.72743977 +0000 UTC m=+69.758834756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5rdtx" (UID: "689dd5e0-863e-4f54-af4b-3c268bfe6351") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:15:47.228018 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.227808    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/689dd5e0-863e-4f54-af4b-3c268bfe6351-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:47.229924 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.229905    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96f53b85-bcf8-4a28-9a05-5738e0e9e873-serving-cert\") pod \"service-ca-operator-5d6c75f84f-llmz9\" (UID: \"96f53b85-bcf8-4a28-9a05-5738e0e9e873\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"
Jul 24 21:15:47.236152 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.236130    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.236534 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.236510    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-749656c945-wnf2b"]
Jul 24 21:15:47.238985 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.238968    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 21:15:47.239195 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.239178    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 24 21:15:47.239303 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.239287    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 24 21:15:47.239415 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.239402    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 21:15:47.239623 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.239608    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2tkzx\""
Jul 24 21:15:47.239740 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.239642    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 24 21:15:47.239740 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.239652    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 24 21:15:47.246863 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.246836    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rht2z\" (UniqueName: \"kubernetes.io/projected/27b45efa-aab7-4a1f-a6fb-3a701fdb7d89-kube-api-access-rht2z\") pod \"network-check-source-7d88cb55d8-bsshq\" (UID: \"27b45efa-aab7-4a1f-a6fb-3a701fdb7d89\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bsshq"
Jul 24 21:15:47.252843 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.252820    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47g9l\" (UniqueName: \"kubernetes.io/projected/96f53b85-bcf8-4a28-9a05-5738e0e9e873-kube-api-access-47g9l\") pod \"service-ca-operator-5d6c75f84f-llmz9\" (UID: \"96f53b85-bcf8-4a28-9a05-5738e0e9e873\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"
Jul 24 21:15:47.253915 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.253893    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lkfs\" (UniqueName: \"kubernetes.io/projected/689dd5e0-863e-4f54-af4b-3c268bfe6351-kube-api-access-9lkfs\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:47.327573 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.327495    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.327573 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.327543    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-stats-auth\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.327573 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.327567    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5gvz\" (UniqueName: \"kubernetes.io/projected/99a5f181-09a0-4f61-85a7-6946878f7980-kube-api-access-w5gvz\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.327808 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.327593    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.327808 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.327767    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-default-certificate\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.368267 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.368234    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:47.377019 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.376991    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bsshq"
Jul 24 21:15:47.396054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.396038    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"
Jul 24 21:15:47.425531 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.425510    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:47.428562 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.428528    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-default-certificate\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.428661 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.428599    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.428661 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.428622    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-stats-auth\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.428661 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.428643    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5gvz\" (UniqueName: \"kubernetes.io/projected/99a5f181-09a0-4f61-85a7-6946878f7980-kube-api-access-w5gvz\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.428819 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.428687    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.428819 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:47.428780    2522 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 21:15:47.428819 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:47.428788    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:47.928769234 +0000 UTC m=+69.960164236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : configmap references non-existent config key: service-ca.crt
Jul 24 21:15:47.428975 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:47.428832    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:47.928816372 +0000 UTC m=+69.960211359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : secret "router-metrics-certs-default" not found
Jul 24 21:15:47.431322 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.431304    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-default-certificate\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.431410 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.431341    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-stats-auth\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.437543 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.437521    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5gvz\" (UniqueName: \"kubernetes.io/projected/99a5f181-09a0-4f61-85a7-6946878f7980-kube-api-access-w5gvz\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.732018 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.731985    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:47.732182 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:47.732133    2522 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:15:47.732221 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:47.732197    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls podName:689dd5e0-863e-4f54-af4b-3c268bfe6351 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:48.732182037 +0000 UTC m=+70.763577022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5rdtx" (UID: "689dd5e0-863e-4f54-af4b-3c268bfe6351") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:15:47.934664 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.934614    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.934664 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:47.934686    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:47.934946 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:47.934820    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:48.934798193 +0000 UTC m=+70.966193182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : configmap references non-existent config key: service-ca.crt
Jul 24 21:15:47.934946 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:47.934851    2522 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 21:15:47.934946 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:47.934893    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:48.934881791 +0000 UTC m=+70.966276794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : secret "router-metrics-certs-default" not found
Jul 24 21:15:48.741907 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:48.741876    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:48.742330 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:48.742023    2522 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:15:48.742330 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:48.742097    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls podName:689dd5e0-863e-4f54-af4b-3c268bfe6351 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:50.742080561 +0000 UTC m=+72.773475568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5rdtx" (UID: "689dd5e0-863e-4f54-af4b-3c268bfe6351") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:15:48.943078 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:48.943042    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:48.943078 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:48.943089    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:48.943279 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:48.943222    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:50.943205059 +0000 UTC m=+72.974600044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : configmap references non-existent config key: service-ca.crt
Jul 24 21:15:48.943279 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:48.943258    2522 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 21:15:48.943372 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:48.943308    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:50.943293944 +0000 UTC m=+72.974688946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : secret "router-metrics-certs-default" not found
Jul 24 21:15:49.094330 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.094235    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:49.195104 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.195074    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:49.353292 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.353222    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:49.454608 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.454581    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:49.776253 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.776206    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9"]
Jul 24 21:15:49.779310 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:15:49.779283    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f53b85_bcf8_4a28_9a05_5738e0e9e873.slice/crio-0d4fd2331ee2c36adb3161098095fa4b3778d2a9d6e14307e3f931e61241c883 WatchSource:0}: Error finding container 0d4fd2331ee2c36adb3161098095fa4b3778d2a9d6e14307e3f931e61241c883: Status 404 returned error can't find the container with id 0d4fd2331ee2c36adb3161098095fa4b3778d2a9d6e14307e3f931e61241c883
Jul 24 21:15:49.795091 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.795066    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-bsshq"]
Jul 24 21:15:49.798809 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:15:49.798786    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b45efa_aab7_4a1f_a6fb_3a701fdb7d89.slice/crio-2e9ddf96d40a0a0cb1b797435430f9c9daa2a2132dc18df0c108b1a7e9d02060 WatchSource:0}: Error finding container 2e9ddf96d40a0a0cb1b797435430f9c9daa2a2132dc18df0c108b1a7e9d02060: Status 404 returned error can't find the container with id 2e9ddf96d40a0a0cb1b797435430f9c9daa2a2132dc18df0c108b1a7e9d02060
Jul 24 21:15:49.818249 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.818218    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9nzl9" event={"ID":"05180a7b-cf89-498d-af4b-6ba6ed49ae0e","Type":"ContainerStarted","Data":"17bb9a777e1294af9f0892560b57cc9ec4ef4e4ec446594e65598820a8b53a90"}
Jul 24 21:15:49.818722 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.818377    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:15:49.819360 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.819337    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9" event={"ID":"96f53b85-bcf8-4a28-9a05-5738e0e9e873","Type":"ContainerStarted","Data":"0d4fd2331ee2c36adb3161098095fa4b3778d2a9d6e14307e3f931e61241c883"}
Jul 24 21:15:49.820892 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.820518    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bsshq" event={"ID":"27b45efa-aab7-4a1f-a6fb-3a701fdb7d89","Type":"ContainerStarted","Data":"2e9ddf96d40a0a0cb1b797435430f9c9daa2a2132dc18df0c108b1a7e9d02060"}
Jul 24 21:15:49.822276 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.822220    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sxqs9" event={"ID":"86301808-2ce5-4813-b527-98b3967b4d15","Type":"ContainerStarted","Data":"7840ee765686ec813b9bf25801a599075552ebbb0b2e96ad894cd2e9fa026684"}
Jul 24 21:15:49.835652 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.835605    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9nzl9" podStartSLOduration=65.767007347 podStartE2EDuration="1m10.835587697s" totalImagesPullingTime="5.06858035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:49.834011546 +0000 UTC m=+71.865406568" watchObservedRunningTime="2026-07-24 21:15:49.835587697 +0000 UTC m=+71.866982706"
Jul 24 21:15:49.849541 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:49.849495    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sxqs9" podStartSLOduration=65.753746155 podStartE2EDuration="1m10.849477176s" totalImagesPullingTime="5.095731021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:49.847970813 +0000 UTC m=+71.879365822" watchObservedRunningTime="2026-07-24 21:15:49.849477176 +0000 UTC m=+71.880872184"
Jul 24 21:15:50.755807 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:50.755769    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:50.755975 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:50.755928    2522 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:15:50.756124 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:50.756007    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls podName:689dd5e0-863e-4f54-af4b-3c268bfe6351 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:54.755986922 +0000 UTC m=+76.787381925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5rdtx" (UID: "689dd5e0-863e-4f54-af4b-3c268bfe6351") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:15:50.827305 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:50.826958    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bsshq" event={"ID":"27b45efa-aab7-4a1f-a6fb-3a701fdb7d89","Type":"ContainerStarted","Data":"16f43a8ca247721230c4ade476f61ed3681b294a04a7be7b2b3e7486f86bfd14"}
Jul 24 21:15:50.842309 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:50.842261    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-bsshq" podStartSLOduration=4.842239713 podStartE2EDuration="4.842239713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:50.841234335 +0000 UTC m=+72.872629342" watchObservedRunningTime="2026-07-24 21:15:50.842239713 +0000 UTC m=+72.873634721"
Jul 24 21:15:50.957568 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:50.957538    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:50.957568 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:50.957573    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:50.957804 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:50.957666    2522 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 21:15:50.957804 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:50.957716    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:54.957697068 +0000 UTC m=+76.989092069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : configmap references non-existent config key: service-ca.crt
Jul 24 21:15:50.957804 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:50.957743    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:54.957732688 +0000 UTC m=+76.989127676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : secret "router-metrics-certs-default" not found
Jul 24 21:15:51.965085 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:51.965052    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sqv7q"]
Jul 24 21:15:51.967999 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:51.967983    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:51.973494 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:51.973328    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 21:15:51.973790 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:51.973772    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 21:15:51.974333 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:51.974309    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 21:15:51.974435 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:51.974341    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v7v2z\""
Jul 24 21:15:51.974435 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:51.974392    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 21:15:51.985620 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:51.985599    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sqv7q"]
Jul 24 21:15:52.020407 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.020386    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:52.066575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.066527    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/043caca9-caf7-46cb-9297-b7be1f5fd9a9-crio-socket\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.066575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.066567    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.066747 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.066660    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.066796 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.066758    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/043caca9-caf7-46cb-9297-b7be1f5fd9a9-data-volume\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.066841 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.066788    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hlsl\" (UniqueName: \"kubernetes.io/projected/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-api-access-4hlsl\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.108781 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.108763    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:52.167735 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.167714    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.167824 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.167799    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.167901 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:52.167845    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:52.167901 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.167869    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/043caca9-caf7-46cb-9297-b7be1f5fd9a9-data-volume\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.167901 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:52.167893    2522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:15:52.168012 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:52.167905    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls podName:043caca9-caf7-46cb-9297-b7be1f5fd9a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:52.667887725 +0000 UTC m=+74.699282731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv7q" (UID: "043caca9-caf7-46cb-9297-b7be1f5fd9a9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:52.168012 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.167925    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hlsl\" (UniqueName: \"kubernetes.io/projected/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-api-access-4hlsl\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.168012 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:52.167933    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm podName:043caca9-caf7-46cb-9297-b7be1f5fd9a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:52.667920064 +0000 UTC m=+74.699315053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sqv7q" (UID: "043caca9-caf7-46cb-9297-b7be1f5fd9a9") : configmap "kube-rbac-proxy" not found
Jul 24 21:15:52.168012 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.167968    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/043caca9-caf7-46cb-9297-b7be1f5fd9a9-crio-socket\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.168186 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.168075    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/043caca9-caf7-46cb-9297-b7be1f5fd9a9-crio-socket\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.168186 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.168150    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/043caca9-caf7-46cb-9297-b7be1f5fd9a9-data-volume\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.177934 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.177918    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hlsl\" (UniqueName: \"kubernetes.io/projected/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-api-access-4hlsl\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.332240 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.332178    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:52.425477 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.425451    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:52.627259 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.627195    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:52.671545 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.671510    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.671639 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.671627    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:52.671730 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:52.671635    2522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:15:52.671730 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:52.671725    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm podName:043caca9-caf7-46cb-9297-b7be1f5fd9a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:53.671705259 +0000 UTC m=+75.703100249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sqv7q" (UID: "043caca9-caf7-46cb-9297-b7be1f5fd9a9") : configmap "kube-rbac-proxy" not found
Jul 24 21:15:52.671841 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:52.671756    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:52.671841 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:52.671809    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls podName:043caca9-caf7-46cb-9297-b7be1f5fd9a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:53.671794428 +0000 UTC m=+75.703189418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv7q" (UID: "043caca9-caf7-46cb-9297-b7be1f5fd9a9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:52.728549 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.728528    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:52.832511 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.832486    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9" event={"ID":"96f53b85-bcf8-4a28-9a05-5738e0e9e873","Type":"ContainerStarted","Data":"5a58934723d4bb618fc708efdf164563282cc5d7c5b953731446b2f0c12c6e93"}
Jul 24 21:15:52.849151 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.849115    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9" podStartSLOduration=4.8251633510000005 podStartE2EDuration="6.849100744s" totalImagesPullingTime="2.023937393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:52.84806869 +0000 UTC m=+74.879463699" watchObservedRunningTime="2026-07-24 21:15:52.849100744 +0000 UTC m=+74.880495752"
Jul 24 21:15:52.868619 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.868582    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:52.968108 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:52.968086    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:53.338752 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:53.338686    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:53.436656 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:53.436630    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:53.676362 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:53.676271    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:53.676495 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:53.676365    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:53.676495 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:53.676435    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:53.676594 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:53.676509    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls podName:043caca9-caf7-46cb-9297-b7be1f5fd9a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:55.676491444 +0000 UTC m=+77.707886431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv7q" (UID: "043caca9-caf7-46cb-9297-b7be1f5fd9a9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:53.676594 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:53.676506    2522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:15:53.676594 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:53.676580    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm podName:043caca9-caf7-46cb-9297-b7be1f5fd9a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:55.676561291 +0000 UTC m=+77.707956322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sqv7q" (UID: "043caca9-caf7-46cb-9297-b7be1f5fd9a9") : configmap "kube-rbac-proxy" not found
Jul 24 21:15:53.767951 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:53.767917    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:54.027629 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:54.027591    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:54.130734 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:54.130711    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:54.171409 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:54.171386    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sw9pw_09ad0063-0c05-496f-9696-cec68aa2aa14/dns-node-resolver/0.log"
Jul 24 21:15:54.784969 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:54.784943    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:15:54.785096 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:54.785040    2522 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:15:54.785096 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:54.785087    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls podName:689dd5e0-863e-4f54-af4b-3c268bfe6351 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:02.78507522 +0000 UTC m=+84.816470207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5rdtx" (UID: "689dd5e0-863e-4f54-af4b-3c268bfe6351") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:15:54.986271 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:54.986238    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:54.986394 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:54.986289    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:15:54.986394 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:54.986366    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:02.986355676 +0000 UTC m=+85.017750663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : configmap references non-existent config key: service-ca.crt
Jul 24 21:15:54.986394 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:54.986368    2522 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 21:15:54.986609 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:54.986418    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:02.98640415 +0000 UTC m=+85.017799135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : secret "router-metrics-certs-default" not found
Jul 24 21:15:55.174474 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:55.174420    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zsccb_b7a9829f-33e2-4b02-a05a-35f9765d9fbb/node-ca/0.log"
Jul 24 21:15:55.665914 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:55.665881    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:55.691773 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:55.691739    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:55.691864 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:55.691832    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:55.691902 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:55.691868    2522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:15:55.691934 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:55.691929    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm podName:043caca9-caf7-46cb-9297-b7be1f5fd9a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:59.691915683 +0000 UTC m=+81.723310673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sqv7q" (UID: "043caca9-caf7-46cb-9297-b7be1f5fd9a9") : configmap "kube-rbac-proxy" not found
Jul 24 21:15:55.691972 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:55.691935    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:55.692012 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:55.691981    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls podName:043caca9-caf7-46cb-9297-b7be1f5fd9a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:59.691970367 +0000 UTC m=+81.723365390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv7q" (UID: "043caca9-caf7-46cb-9297-b7be1f5fd9a9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:55.772057 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:55.772031    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j7h2p_7ff7824c-d44f-48f1-af97-518f5b3b5d9b/serve-healthcheck-canary/0.log"
Jul 24 21:15:56.573037 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:56.573000    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:59.517602 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:59.517562    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:59.616065 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:59.616030    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:59.720812 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:59.720776    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:59.720964 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:59.720846    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:15:59.720964 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:59.720950    2522 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:15:59.720964 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:59.720955    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:59.721105 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:59.721009    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm podName:043caca9-caf7-46cb-9297-b7be1f5fd9a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:07.720993627 +0000 UTC m=+89.752388613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sqv7q" (UID: "043caca9-caf7-46cb-9297-b7be1f5fd9a9") : configmap "kube-rbac-proxy" not found
Jul 24 21:15:59.721105 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:15:59.721024    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls podName:043caca9-caf7-46cb-9297-b7be1f5fd9a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:07.721017002 +0000 UTC m=+89.752411987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv7q" (UID: "043caca9-caf7-46cb-9297-b7be1f5fd9a9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:59.814468 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:15:59.814314    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:00.075972 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:00.075889    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:00.231605 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:00.231573    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:00.389323 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:00.389228    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:00.487455 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:00.487422    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:00.566650 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:00.566627    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:00.720951 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:00.720918    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:01.508393 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:01.508364    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:01.919371 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:01.919284    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:02.348233 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:02.348194    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:02.511639 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:02.511608    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:02.844575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:02.844541    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:16:02.844802 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:02.844742    2522 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:02.844869 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:02.844845    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls podName:689dd5e0-863e-4f54-af4b-3c268bfe6351 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:18.844819164 +0000 UTC m=+100.876214218 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5rdtx" (UID: "689dd5e0-863e-4f54-af4b-3c268bfe6351") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:02.905994 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:02.905960    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:03.047132 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:03.047111    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:16:03.047453 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:03.047152    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:16:03.047453 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:03.047256    2522 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 24 21:16:03.047453 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:03.047268    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:19.047254827 +0000 UTC m=+101.078649813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : configmap references non-existent config key: service-ca.crt
Jul 24 21:16:03.047453 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:03.047303    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs podName:99a5f181-09a0-4f61-85a7-6946878f7980 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:19.047290578 +0000 UTC m=+101.078685565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs") pod "router-default-749656c945-wnf2b" (UID: "99a5f181-09a0-4f61-85a7-6946878f7980") : secret "router-metrics-certs-default" not found
Jul 24 21:16:03.322149 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:03.322109    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:03.516971 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:03.516916    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:03.615066 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:03.614987    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:03.900216 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:03.900140    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:04.311466 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:04.311436    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:04.700589 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:04.700561    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:04.902139 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:04.902106    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:05.229133 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:05.229101    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:05.505269 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:05.505171    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:05.614954 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:05.614926    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:05.904718 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:05.904622    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:06.299260 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:06.299229    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:06.502577 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:06.502543    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:07.106997 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:07.106959    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:07.699308 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:07.699281    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:07.767180 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:07.767148    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:07.787037 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:07.787016    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:16:07.787126 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:07.787088    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:16:07.787219 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:07.787201    2522 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:07.787292 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:07.787280    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls podName:043caca9-caf7-46cb-9297-b7be1f5fd9a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:23.78725786 +0000 UTC m=+105.818652871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sqv7q" (UID: "043caca9-caf7-46cb-9297-b7be1f5fd9a9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:07.787614 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:07.787597    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/043caca9-caf7-46cb-9297-b7be1f5fd9a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:16:07.901117 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:07.901090    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:08.102582 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:08.102511    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:08.368227 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:08.368120    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:08.932780 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:08.932721    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.033066 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:09.033032    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.135521 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:09.135495    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.238961 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:09.238935    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.414595 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:09.414564    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.515576 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:09.515507    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.617575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:09.617544    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.716733 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:09.716706    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.817596 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:09.817529    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.917815 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:09.917792    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:10.018265 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:10.018241    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:10.118462 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:10.118401    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:10.181193 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:10.181152    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:10.298888 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:10.298868    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:10.415894 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:10.415823    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:10.627050 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:10.627021    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:10.737237 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:10.737196    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:11.840372 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:11.840324    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:11.903402 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:11.903364    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:17.798859 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:17.798820    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:17.912498 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:17.912460    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:18.105708 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.105620    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:18.164889 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.164864    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:16:18.165021 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.164914    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:16:18.167333 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.167310    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e21e54d-fb5e-4a9a-9f64-908449a5e755-metrics-tls\") pod \"dns-default-krwsr\" (UID: \"1e21e54d-fb5e-4a9a-9f64-908449a5e755\") " pod="openshift-dns/dns-default-krwsr"
Jul 24 21:16:18.167467 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.167446    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") pod \"image-registry-67c49bbfb6-9t74k\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") " pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:16:18.213739 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.213717    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:18.430338 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.430280    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rzr52\""
Jul 24 21:16:18.438637 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.438615    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:16:18.444924 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.444904    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wbvbs\""
Jul 24 21:16:18.452830 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.452810    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-krwsr"
Jul 24 21:16:18.582312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.582282    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67c49bbfb6-9t74k"]
Jul 24 21:16:18.586093 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:18.586071    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104d73c0_ff20_47b8_9774_6b3eaa2708ae.slice/crio-0c96e3f7678bbcc59cacaf969454504b83e629db578c819b597d4a7e0e0777d4 WatchSource:0}: Error finding container 0c96e3f7678bbcc59cacaf969454504b83e629db578c819b597d4a7e0e0777d4: Status 404 returned error can't find the container with id 0c96e3f7678bbcc59cacaf969454504b83e629db578c819b597d4a7e0e0777d4
Jul 24 21:16:18.595635 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.595614    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-krwsr"]
Jul 24 21:16:18.597453 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:18.597432    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21e54d_fb5e_4a9a_9f64_908449a5e755.slice/crio-c9a5cc58d0732c30bfc14e9ace067dd60944c77c4bf6ffaae4d6d9c9cf62c6c5 WatchSource:0}: Error finding container c9a5cc58d0732c30bfc14e9ace067dd60944c77c4bf6ffaae4d6d9c9cf62c6c5: Status 404 returned error can't find the container with id c9a5cc58d0732c30bfc14e9ace067dd60944c77c4bf6ffaae4d6d9c9cf62c6c5
Jul 24 21:16:18.870490 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.870461    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:16:18.872829 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.872809    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/689dd5e0-863e-4f54-af4b-3c268bfe6351-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5rdtx\" (UID: \"689dd5e0-863e-4f54-af4b-3c268bfe6351\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:16:18.906103 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.906077    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-krwsr" event={"ID":"1e21e54d-fb5e-4a9a-9f64-908449a5e755","Type":"ContainerStarted","Data":"c9a5cc58d0732c30bfc14e9ace067dd60944c77c4bf6ffaae4d6d9c9cf62c6c5"}
Jul 24 21:16:18.907604 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.907580    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k" event={"ID":"104d73c0-ff20-47b8-9774-6b3eaa2708ae","Type":"ContainerStarted","Data":"c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71"}
Jul 24 21:16:18.907730 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.907620    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k" event={"ID":"104d73c0-ff20-47b8-9774-6b3eaa2708ae","Type":"ContainerStarted","Data":"0c96e3f7678bbcc59cacaf969454504b83e629db578c819b597d4a7e0e0777d4"}
Jul 24 21:16:18.907730 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.907712    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:16:18.939861 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:18.939813    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k" podStartSLOduration=99.939799051 podStartE2EDuration="1m39.939799051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:18.939288948 +0000 UTC m=+100.970683960" watchObservedRunningTime="2026-07-24 21:16:18.939799051 +0000 UTC m=+100.971194058"
Jul 24 21:16:19.072561 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.072519    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:16:19.072754 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.072569    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:16:19.073369 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.073328    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99a5f181-09a0-4f61-85a7-6946878f7980-service-ca-bundle\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:16:19.075343 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.075322    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99a5f181-09a0-4f61-85a7-6946878f7980-metrics-certs\") pod \"router-default-749656c945-wnf2b\" (UID: \"99a5f181-09a0-4f61-85a7-6946878f7980\") " pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:16:19.162341 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.162275    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"
Jul 24 21:16:19.315457 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.315426    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx"]
Jul 24 21:16:19.319094 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:19.319059    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689dd5e0_863e_4f54_af4b_3c268bfe6351.slice/crio-4e017ad72cbc9fd58d010a1bcef0a2e01edea4b0af84e1b8604821a73107cfa8 WatchSource:0}: Error finding container 4e017ad72cbc9fd58d010a1bcef0a2e01edea4b0af84e1b8604821a73107cfa8: Status 404 returned error can't find the container with id 4e017ad72cbc9fd58d010a1bcef0a2e01edea4b0af84e1b8604821a73107cfa8
Jul 24 21:16:19.354623 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.354603    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:16:19.501406 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.501374    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-749656c945-wnf2b"]
Jul 24 21:16:19.504527 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:19.504493    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a5f181_09a0_4f61_85a7_6946878f7980.slice/crio-acdd368ed0f57d037fd6a861cda3b4b514a6106b028d8031aa5693679c66d254 WatchSource:0}: Error finding container acdd368ed0f57d037fd6a861cda3b4b514a6106b028d8031aa5693679c66d254: Status 404 returned error can't find the container with id acdd368ed0f57d037fd6a861cda3b4b514a6106b028d8031aa5693679c66d254
Jul 24 21:16:19.913684 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.913584    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-749656c945-wnf2b" event={"ID":"99a5f181-09a0-4f61-85a7-6946878f7980","Type":"ContainerStarted","Data":"edb9b6e49cfe77d932dfb1a177252cbe22974c02bf472a412ef18194494c7df0"}
Jul 24 21:16:19.913684 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.913645    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-749656c945-wnf2b" event={"ID":"99a5f181-09a0-4f61-85a7-6946878f7980","Type":"ContainerStarted","Data":"acdd368ed0f57d037fd6a861cda3b4b514a6106b028d8031aa5693679c66d254"}
Jul 24 21:16:19.915169 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.915141    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx" event={"ID":"689dd5e0-863e-4f54-af4b-3c268bfe6351","Type":"ContainerStarted","Data":"4e017ad72cbc9fd58d010a1bcef0a2e01edea4b0af84e1b8604821a73107cfa8"}
Jul 24 21:16:19.921624 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.921602    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55db5f7fb-kxzsx"]
Jul 24 21:16:19.924650 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.924620    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:19.956518 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.956482    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 21:16:19.956518 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.956487    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 21:16:19.956704 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.956554    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 21:16:19.956704 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.956575    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 21:16:19.960629 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.960612    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 21:16:19.960888 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.960874    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 21:16:19.981832 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.981811    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 21:16:19.981996 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.981982    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c84kp\""
Jul 24 21:16:19.983057 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.983011    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-749656c945-wnf2b" podStartSLOduration=32.982993117 podStartE2EDuration="32.982993117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:19.982216583 +0000 UTC m=+102.013611591" watchObservedRunningTime="2026-07-24 21:16:19.982993117 +0000 UTC m=+102.014388123"
Jul 24 21:16:19.992205 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:19.992168    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55db5f7fb-kxzsx"]
Jul 24 21:16:20.082066 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.082035    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-oauth-config\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.082202 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.082096    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-serving-cert\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.082358 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.082340    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-oauth-serving-cert\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.082491 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.082461    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-service-ca\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.082716 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.082699    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-config\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.082805 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.082744    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djkbk\" (UniqueName: \"kubernetes.io/projected/061e4c22-f3b9-4b63-a7ec-78d2154fc989-kube-api-access-djkbk\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.183219 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.183156    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-service-ca\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.183349 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.183234    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-config\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.183349 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.183259    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkbk\" (UniqueName: \"kubernetes.io/projected/061e4c22-f3b9-4b63-a7ec-78d2154fc989-kube-api-access-djkbk\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.183349 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.183302    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-oauth-config\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.183349 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.183328    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-serving-cert\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.183548 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.183352    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-oauth-serving-cert\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.183966 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.183939    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-service-ca\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.184075 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.183971    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-config\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.184311 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.184288    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-oauth-serving-cert\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.186246 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.186222    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-serving-cert\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.186348 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.186298    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-oauth-config\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.193273 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.193213    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkbk\" (UniqueName: \"kubernetes.io/projected/061e4c22-f3b9-4b63-a7ec-78d2154fc989-kube-api-access-djkbk\") pod \"console-55db5f7fb-kxzsx\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") " pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.235809 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.235788    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:20.355618 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.355556    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:16:20.358839 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.358815    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:16:20.467755 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.467730    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55db5f7fb-kxzsx"]
Jul 24 21:16:20.473397 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:20.473374    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061e4c22_f3b9_4b63_a7ec_78d2154fc989.slice/crio-eb15470a375561950f253589546abcda2622c0a83c2b8f3bf673ba87545a695b WatchSource:0}: Error finding container eb15470a375561950f253589546abcda2622c0a83c2b8f3bf673ba87545a695b: Status 404 returned error can't find the container with id eb15470a375561950f253589546abcda2622c0a83c2b8f3bf673ba87545a695b
Jul 24 21:16:20.830239 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.830164    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9nzl9"
Jul 24 21:16:20.925781 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.925748    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-krwsr" event={"ID":"1e21e54d-fb5e-4a9a-9f64-908449a5e755","Type":"ContainerStarted","Data":"8bdd264044af6f0303233213331d4e376e639784378849340f612b76bcba014c"}
Jul 24 21:16:20.926214 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.925834    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-krwsr" event={"ID":"1e21e54d-fb5e-4a9a-9f64-908449a5e755","Type":"ContainerStarted","Data":"1656fdc443d3681dc16e7565af24e19387036e714418032b54eaec39ce03b38c"}
Jul 24 21:16:20.926214 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.925985    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-krwsr"
Jul 24 21:16:20.927218 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.927187    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55db5f7fb-kxzsx" event={"ID":"061e4c22-f3b9-4b63-a7ec-78d2154fc989","Type":"ContainerStarted","Data":"eb15470a375561950f253589546abcda2622c0a83c2b8f3bf673ba87545a695b"}
Jul 24 21:16:20.927491 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.927469    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:16:20.929029 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.928997    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-749656c945-wnf2b"
Jul 24 21:16:20.946065 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:20.946017    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-krwsr" podStartSLOduration=65.172360999 podStartE2EDuration="1m6.946000852s" totalImagesPullingTime="1.773639853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:20.945828752 +0000 UTC m=+102.977223760" watchObservedRunningTime="2026-07-24 21:16:20.946000852 +0000 UTC m=+102.977395862"
Jul 24 21:16:21.872195 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:21.872138    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:21.932519 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:21.932481    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx" event={"ID":"689dd5e0-863e-4f54-af4b-3c268bfe6351","Type":"ContainerStarted","Data":"a90ac1047425ff774fe51f5aac78fbaccd3548f25c46375db72ae8f9b69e6705"}
Jul 24 21:16:21.954068 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:21.954011    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5rdtx" podStartSLOduration=33.869521606 podStartE2EDuration="35.953992777s" totalImagesPullingTime="2.084471171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:21.953541297 +0000 UTC m=+103.984936306" watchObservedRunningTime="2026-07-24 21:16:21.953992777 +0000 UTC m=+103.985387786"
Jul 24 21:16:21.968011 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:21.967987    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:22.051554 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.051524    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:22.066615 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.065429    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd"]
Jul 24 21:16:22.082719 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.082646    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd"]
Jul 24 21:16:22.082844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.082800    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd"
Jul 24 21:16:22.085873 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.085832    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gpj4l\""
Jul 24 21:16:22.086004 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.085979    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 24 21:16:22.202210 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.202182    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c07971e1-0f44-4fcd-8331-c5efe52c3fc1-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-hrnqd\" (UID: \"c07971e1-0f44-4fcd-8331-c5efe52c3fc1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd"
Jul 24 21:16:22.302986 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.302955    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c07971e1-0f44-4fcd-8331-c5efe52c3fc1-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-hrnqd\" (UID: \"c07971e1-0f44-4fcd-8331-c5efe52c3fc1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd"
Jul 24 21:16:22.303121 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:22.303095    2522 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 24 21:16:22.303181 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:22.303169    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c07971e1-0f44-4fcd-8331-c5efe52c3fc1-tls-certificates podName:c07971e1-0f44-4fcd-8331-c5efe52c3fc1 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:22.803152922 +0000 UTC m=+104.834547920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c07971e1-0f44-4fcd-8331-c5efe52c3fc1-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-hrnqd" (UID: "c07971e1-0f44-4fcd-8331-c5efe52c3fc1") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 24 21:16:22.492936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.492865    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:22.807964 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.807880    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c07971e1-0f44-4fcd-8331-c5efe52c3fc1-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-hrnqd\" (UID: \"c07971e1-0f44-4fcd-8331-c5efe52c3fc1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd"
Jul 24 21:16:22.810813 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.810784    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c07971e1-0f44-4fcd-8331-c5efe52c3fc1-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-hrnqd\" (UID: \"c07971e1-0f44-4fcd-8331-c5efe52c3fc1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd"
Jul 24 21:16:22.994612 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:22.994581    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd"
Jul 24 21:16:23.648138 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:23.648101    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:23.804342 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:23.804265    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:23.818417 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:23.818386    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:16:23.821031 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:23.821010    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/043caca9-caf7-46cb-9297-b7be1f5fd9a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sqv7q\" (UID: \"043caca9-caf7-46cb-9297-b7be1f5fd9a9\") " pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:16:24.076589 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:24.076551    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sqv7q"
Jul 24 21:16:24.231987 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:24.231962    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd"]
Jul 24 21:16:24.233221 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:24.233190    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07971e1_0f44_4fcd_8331_c5efe52c3fc1.slice/crio-9bd40b128b8ccb7333da92e770f3cae62e15289f0d54d5a9de4caca2d072a343 WatchSource:0}: Error finding container 9bd40b128b8ccb7333da92e770f3cae62e15289f0d54d5a9de4caca2d072a343: Status 404 returned error can't find the container with id 9bd40b128b8ccb7333da92e770f3cae62e15289f0d54d5a9de4caca2d072a343
Jul 24 21:16:24.250750 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:24.250729    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sqv7q"]
Jul 24 21:16:24.253934 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:24.253914    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043caca9_caf7_46cb_9297_b7be1f5fd9a9.slice/crio-6e367960b497ce67043b8ac1848889517eceb49e02d0dc8c307d2e47d5e3e802 WatchSource:0}: Error finding container 6e367960b497ce67043b8ac1848889517eceb49e02d0dc8c307d2e47d5e3e802: Status 404 returned error can't find the container with id 6e367960b497ce67043b8ac1848889517eceb49e02d0dc8c307d2e47d5e3e802
Jul 24 21:16:24.814566 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:24.814475    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:24.955022 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:24.954178    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55db5f7fb-kxzsx" event={"ID":"061e4c22-f3b9-4b63-a7ec-78d2154fc989","Type":"ContainerStarted","Data":"6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760"}
Jul 24 21:16:24.955022 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:24.954265    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:24.956778 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:24.956751    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sqv7q" event={"ID":"043caca9-caf7-46cb-9297-b7be1f5fd9a9","Type":"ContainerStarted","Data":"6e367960b497ce67043b8ac1848889517eceb49e02d0dc8c307d2e47d5e3e802"}
Jul 24 21:16:24.958394 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:24.958349    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd" event={"ID":"c07971e1-0f44-4fcd-8331-c5efe52c3fc1","Type":"ContainerStarted","Data":"9bd40b128b8ccb7333da92e770f3cae62e15289f0d54d5a9de4caca2d072a343"}
Jul 24 21:16:25.070024 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:25.069962    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:25.141319 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:25.141277    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:25.963523 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:25.963480    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd" event={"ID":"c07971e1-0f44-4fcd-8331-c5efe52c3fc1","Type":"ContainerStarted","Data":"bb6fda20a799dc4e26a9643c79985eb301bfc432131058d566e14a20102437df"}
Jul 24 21:16:25.980990 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:25.980934    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd" podStartSLOduration=2.448576611 podStartE2EDuration="3.980916326s" totalImagesPullingTime="1.532339715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:25.980080593 +0000 UTC m=+108.011475603" watchObservedRunningTime="2026-07-24 21:16:25.980916326 +0000 UTC m=+108.012311335"
Jul 24 21:16:25.981322 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:25.981288    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55db5f7fb-kxzsx" podStartSLOduration=2.981064152 podStartE2EDuration="6.981276587s" totalImagesPullingTime="4.000212435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:24.983280396 +0000 UTC m=+107.014675414" watchObservedRunningTime="2026-07-24 21:16:25.981276587 +0000 UTC m=+108.012671598"
Jul 24 21:16:26.690880 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:26.690837    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:26.758720 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:26.758664    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:26.823504 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:26.823465    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:26.967844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:26.967760    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd"
Jul 24 21:16:26.973105 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:26.973082    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hrnqd"
Jul 24 21:16:27.086982 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.086945    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:27.127578 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.127551    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-74kgg"]
Jul 24 21:16:27.137136 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.137110    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.139531 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.139505    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 21:16:27.139880 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.139855    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-74kgg"]
Jul 24 21:16:27.139987 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.139964    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 21:16:27.139987 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.139980    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 21:16:27.140227 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.140209    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5225f\""
Jul 24 21:16:27.187926 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.187879    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:27.243585 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.243519    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:27.246445 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.246424    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.246558 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.246476    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.246623 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.246556    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dkk\" (UniqueName: \"kubernetes.io/projected/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-kube-api-access-r5dkk\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.246693 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.246654    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.348042 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.348008    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dkk\" (UniqueName: \"kubernetes.io/projected/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-kube-api-access-r5dkk\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.348183 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.348077    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.348183 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.348138    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.348291 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.348180    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.349026 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.348923    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.350973 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.350947    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.351074 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.350980    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.358519 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.358494    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dkk\" (UniqueName: \"kubernetes.io/projected/d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b-kube-api-access-r5dkk\") pod \"prometheus-operator-84d69f794d-74kgg\" (UID: \"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.448473 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.448438    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg"
Jul 24 21:16:27.582144 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.582068    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:27.684313 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:27.684270    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:28.344509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.344471    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:28.447002 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.446970    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:28.650940 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.650871    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7489dc58b6-4q5tb"]
Jul 24 21:16:28.655428 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.655396    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:28.657390 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.657375    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.666531 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.666507    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 24 21:16:28.667692 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.667655    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7489dc58b6-4q5tb"]
Jul 24 21:16:28.723659 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.723628    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:28.759055 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.759026    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-service-ca\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.759192 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.759096    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-serving-cert\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.759192 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.759120    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-oauth-serving-cert\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.759192 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.759137    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-oauth-config\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.759192 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.759162    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5xg2\" (UniqueName: \"kubernetes.io/projected/20d225c2-33d2-4f52-ab38-27848ffc551f-kube-api-access-h5xg2\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.759192 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.759187    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-console-config\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.759506 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.759285    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-trusted-ca-bundle\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.860733 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.860693    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-trusted-ca-bundle\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.860896 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.860769    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-service-ca\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.860896 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.860882    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-serving-cert\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.861007 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.860919    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-oauth-serving-cert\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.861007 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.860957    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-oauth-config\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.861007 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.861001    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5xg2\" (UniqueName: \"kubernetes.io/projected/20d225c2-33d2-4f52-ab38-27848ffc551f-kube-api-access-h5xg2\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.861150 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.861037    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-console-config\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.861735 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.861653    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-service-ca\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.861838 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.861758    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-oauth-serving-cert\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.861838 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.861785    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-console-config\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.862211 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.862183    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-trusted-ca-bundle\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.863919 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.863891    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-serving-cert\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.863919 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.863908    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-oauth-config\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.873983 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.873962    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5xg2\" (UniqueName: \"kubernetes.io/projected/20d225c2-33d2-4f52-ab38-27848ffc551f-kube-api-access-h5xg2\") pod \"console-7489dc58b6-4q5tb\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") " pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:28.968113 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:28.968085    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:29.241456 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:29.241431    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7489dc58b6-4q5tb"]
Jul 24 21:16:29.245093 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:29.245071    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d225c2_33d2_4f52_ab38_27848ffc551f.slice/crio-f5c276d2f201b7c0648ecab11818a3be12c9849c13ea6d233466a2ecc76c4d65 WatchSource:0}: Error finding container f5c276d2f201b7c0648ecab11818a3be12c9849c13ea6d233466a2ecc76c4d65: Status 404 returned error can't find the container with id f5c276d2f201b7c0648ecab11818a3be12c9849c13ea6d233466a2ecc76c4d65
Jul 24 21:16:29.262050 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:29.262025    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-74kgg"]
Jul 24 21:16:29.265223 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:29.265192    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6299c5f_caa4_4b2d_ac55_a6d0c50ac34b.slice/crio-94917bdd0d5f673bd6d256b568dc0f0a6715ebc11a828ccb26f4b167de87ab66 WatchSource:0}: Error finding container 94917bdd0d5f673bd6d256b568dc0f0a6715ebc11a828ccb26f4b167de87ab66: Status 404 returned error can't find the container with id 94917bdd0d5f673bd6d256b568dc0f0a6715ebc11a828ccb26f4b167de87ab66
Jul 24 21:16:29.851853 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:29.851763    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:29.919710 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:29.919640    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:29.982446 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:29.982410    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7489dc58b6-4q5tb" event={"ID":"20d225c2-33d2-4f52-ab38-27848ffc551f","Type":"ContainerStarted","Data":"4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd"}
Jul 24 21:16:29.982613 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:29.982475    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7489dc58b6-4q5tb" event={"ID":"20d225c2-33d2-4f52-ab38-27848ffc551f","Type":"ContainerStarted","Data":"f5c276d2f201b7c0648ecab11818a3be12c9849c13ea6d233466a2ecc76c4d65"}
Jul 24 21:16:29.984860 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:29.984833    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg" event={"ID":"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b","Type":"ContainerStarted","Data":"94917bdd0d5f673bd6d256b568dc0f0a6715ebc11a828ccb26f4b167de87ab66"}
Jul 24 21:16:29.986631 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:29.986606    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sqv7q" event={"ID":"043caca9-caf7-46cb-9297-b7be1f5fd9a9","Type":"ContainerStarted","Data":"ce7be8713c656d59041c516b2085601d93dee0ffc24758229a3225eea910eaa6"}
Jul 24 21:16:30.236899 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:30.236865    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:30.237068 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:30.236929    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:30.242749 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:30.242723    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:30.264442 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:30.264391    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7489dc58b6-4q5tb" podStartSLOduration=2.264375422 podStartE2EDuration="2.264375422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:30.005698888 +0000 UTC m=+112.037093889" watchObservedRunningTime="2026-07-24 21:16:30.264375422 +0000 UTC m=+112.295770431"
Jul 24 21:16:30.934845 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:30.934763    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-krwsr"
Jul 24 21:16:30.956629 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:30.956591    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:31.000081 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:31.000041    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg" event={"ID":"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b","Type":"ContainerStarted","Data":"de7a6744eb9394ff57c375a680dc2ee612be18c5a0d781b1d2afad197b591b7c"}
Jul 24 21:16:31.000251 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:31.000143    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg" event={"ID":"d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b","Type":"ContainerStarted","Data":"5f03f0e197d2b8a04941e2ce53cdf2ee24c746c2af905ddf49eefa707b99bb1b"}
Jul 24 21:16:31.004729 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:31.004708    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:16:31.022801 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:31.022749    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-74kgg" podStartSLOduration=2.624204509 podStartE2EDuration="4.022735326s" totalImagesPullingTime="1.398530817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:31.021925982 +0000 UTC m=+113.053321001" watchObservedRunningTime="2026-07-24 21:16:31.022735326 +0000 UTC m=+113.054130328"
Jul 24 21:16:32.296084 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.296043    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:32.603139 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.603031    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:32.628458 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.628429    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"]
Jul 24 21:16:32.641438 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.641416    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.645051 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.645027    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 24 21:16:32.645247 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.645233    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9bjd2\""
Jul 24 21:16:32.645472 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.645455    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 24 21:16:32.653773 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.653756    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"]
Jul 24 21:16:32.672977 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.672949    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b4zps"]
Jul 24 21:16:32.676470 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.676454    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.679938 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.679920    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 21:16:32.680581 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.680551    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 21:16:32.680581 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.680569    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 21:16:32.680728 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.680612    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p42fm\""
Jul 24 21:16:32.695064 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.695041    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.695164 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.695099    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f84v\" (UniqueName: \"kubernetes.io/projected/0a96bf67-2b22-4d4d-8486-3f01d422d89b-kube-api-access-8f84v\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.695220 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.695191    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a96bf67-2b22-4d4d-8486-3f01d422d89b-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.695278 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.695230    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.701853 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.701834    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"]
Jul 24 21:16:32.706713 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.706695    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.709482 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.709419    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 24 21:16:32.709815 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.709797    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5g2f6\""
Jul 24 21:16:32.709909 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.709832    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 24 21:16:32.709909 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.709849    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 24 21:16:32.725375 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.725354    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"]
Jul 24 21:16:32.752126 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.752096    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:32.796018 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.795988    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.796154 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796027    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjl5\" (UniqueName: \"kubernetes.io/projected/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-kube-api-access-mpjl5\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.796154 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796106    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd7xx\" (UniqueName: \"kubernetes.io/projected/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-api-access-fd7xx\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.796269 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796174    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.796269 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796209    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-sys\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.796353 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796269    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.796353 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796297    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f84v\" (UniqueName: \"kubernetes.io/projected/0a96bf67-2b22-4d4d-8486-3f01d422d89b-kube-api-access-8f84v\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.796353 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796337    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-metrics-client-ca\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.796476 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796383    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.796476 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:32.796392    2522 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 24 21:16:32.796476 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796407    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-tls\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.796476 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796446    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-textfile\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.796662 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:32.796495    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-tls podName:0a96bf67-2b22-4d4d-8486-3f01d422d89b nodeName:}" failed. No retries permitted until 2026-07-24 21:16:33.296477079 +0000 UTC m=+115.327872065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-drrpz" (UID: "0a96bf67-2b22-4d4d-8486-3f01d422d89b") : secret "openshift-state-metrics-tls" not found
Jul 24 21:16:32.796662 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796513    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.796662 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796544    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-root\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.796662 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796592    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a96bf67-2b22-4d4d-8486-3f01d422d89b-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.796662 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796623    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0fc70724-9e58-4b11-9c34-3b9563dfd847-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.796662 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796654    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fc70724-9e58-4b11-9c34-3b9563dfd847-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.796917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796715    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.796917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796742    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-wtmp\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.796917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.796770    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.797318 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.797293    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a96bf67-2b22-4d4d-8486-3f01d422d89b-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.799342 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.799313    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.806395 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.806372    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f84v\" (UniqueName: \"kubernetes.io/projected/0a96bf67-2b22-4d4d-8486-3f01d422d89b-kube-api-access-8f84v\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:32.886213 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.886189    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:32.897601 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.897581    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0fc70724-9e58-4b11-9c34-3b9563dfd847-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.897724 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.897607    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fc70724-9e58-4b11-9c34-3b9563dfd847-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.897724 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.897634    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-wtmp\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.897724 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.897650    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.897724 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.897692    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.897724 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.897723    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjl5\" (UniqueName: \"kubernetes.io/projected/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-kube-api-access-mpjl5\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.897980 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.897765    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd7xx\" (UniqueName: \"kubernetes.io/projected/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-api-access-fd7xx\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.897980 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.897801    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-wtmp\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.897980 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.897959    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898128 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.897997    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-sys\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898128 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898050    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0fc70724-9e58-4b11-9c34-3b9563dfd847-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.898128 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898087    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-sys\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898128 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898105    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-metrics-client-ca\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898143    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.898312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898160    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-tls\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898192    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-textfile\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898217    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898312 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898245    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-root\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898549 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898339    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-root\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898549 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:32.898415    2522 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 21:16:32.898549 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898420    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fc70724-9e58-4b11-9c34-3b9563dfd847-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.898549 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:32.898465    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-tls podName:e5ea6b64-385e-40a4-84fb-bc4666e8c2a3 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:33.398446784 +0000 UTC m=+115.429841774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-tls") pod "node-exporter-b4zps" (UID: "e5ea6b64-385e-40a4-84fb-bc4666e8c2a3") : secret "node-exporter-tls" not found
Jul 24 21:16:32.898778 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898752    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-textfile\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898872 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898851    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898911 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898863    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-metrics-client-ca\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.898911 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.898889    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.900224 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.900206    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.900416 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.900397    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.900664 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.900647    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.908382 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.908364    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd7xx\" (UniqueName: \"kubernetes.io/projected/0fc70724-9e58-4b11-9c34-3b9563dfd847-kube-api-access-fd7xx\") pod \"kube-state-metrics-56d9f944b4-sznnf\" (UID: \"0fc70724-9e58-4b11-9c34-3b9563dfd847\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:32.908640 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.908623    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjl5\" (UniqueName: \"kubernetes.io/projected/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-kube-api-access-mpjl5\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:32.998841 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:32.998813    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:33.010539 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.010510    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sqv7q" event={"ID":"043caca9-caf7-46cb-9297-b7be1f5fd9a9","Type":"ContainerStarted","Data":"9f78763dd4b6f97b1c0d39f59602d63a6153ff4d2b6c89a5df4a5ffea4c91903"}
Jul 24 21:16:33.015624 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.015605    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"
Jul 24 21:16:33.097709 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.097680    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:33.141811 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.141750    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf"]
Jul 24 21:16:33.144495 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:33.144467    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc70724_9e58_4b11_9c34_3b9563dfd847.slice/crio-c4afec9ba459ab7130912525cf9bf382a085c21a359067175077aa8f07858073 WatchSource:0}: Error finding container c4afec9ba459ab7130912525cf9bf382a085c21a359067175077aa8f07858073: Status 404 returned error can't find the container with id c4afec9ba459ab7130912525cf9bf382a085c21a359067175077aa8f07858073
Jul 24 21:16:33.199002 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.198828    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:33.301655 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.301628    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:33.301978 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:33.301760    2522 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 24 21:16:33.301978 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:33.301820    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-tls podName:0a96bf67-2b22-4d4d-8486-3f01d422d89b nodeName:}" failed. No retries permitted until 2026-07-24 21:16:34.301804747 +0000 UTC m=+116.333199736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-drrpz" (UID: "0a96bf67-2b22-4d4d-8486-3f01d422d89b") : secret "openshift-state-metrics-tls" not found
Jul 24 21:16:33.353304 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.353270    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:33.402696 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.402627    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-tls\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:33.404881 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.404864    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5ea6b64-385e-40a4-84fb-bc4666e8c2a3-node-exporter-tls\") pod \"node-exporter-b4zps\" (UID: \"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3\") " pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:33.479229 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.479200    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:33.586564 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.586538    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b4zps"
Jul 24 21:16:33.587012 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.586977    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:33.598596 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:33.598568    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ea6b64_385e_40a4_84fb_bc4666e8c2a3.slice/crio-c9ded99dbc5d8943efb43f789faec9c2e55242097a48de3696ff4aea75db8b5e WatchSource:0}: Error finding container c9ded99dbc5d8943efb43f789faec9c2e55242097a48de3696ff4aea75db8b5e: Status 404 returned error can't find the container with id c9ded99dbc5d8943efb43f789faec9c2e55242097a48de3696ff4aea75db8b5e
Jul 24 21:16:33.694192 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.694158    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:33.694633 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.694610    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:16:33.699585 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.699566    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.703621 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.703594    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 24 21:16:33.703750 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.703635    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 24 21:16:33.703750 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.703638    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 24 21:16:33.704102 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.704054    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 24 21:16:33.704102 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.704074    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 24 21:16:33.704102 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.704084    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 24 21:16:33.704308 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.704292    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 24 21:16:33.704368 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.704311    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7dbvh\""
Jul 24 21:16:33.704415 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.704408    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 24 21:16:33.704577 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.704531    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 24 21:16:33.729205 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.729182    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:16:33.806046 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806016    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-config-volume\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806179 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806068    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806179 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806125    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806179 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806153    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/69cd2ae2-6450-4396-8ab8-97075bf3978f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806179 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806173    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/69cd2ae2-6450-4396-8ab8-97075bf3978f-config-out\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806472 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806251    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69cd2ae2-6450-4396-8ab8-97075bf3978f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806472 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806294    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806472 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806340    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktw4s\" (UniqueName: \"kubernetes.io/projected/69cd2ae2-6450-4396-8ab8-97075bf3978f-kube-api-access-ktw4s\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806472 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806368    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/69cd2ae2-6450-4396-8ab8-97075bf3978f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806472 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806397    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806472 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806420    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806472 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806453    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69cd2ae2-6450-4396-8ab8-97075bf3978f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.806825 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.806514    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-web-config\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907302 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907269    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-config-volume\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907494 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907321    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907494 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907376    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907494 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907407    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/69cd2ae2-6450-4396-8ab8-97075bf3978f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907494 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907430    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/69cd2ae2-6450-4396-8ab8-97075bf3978f-config-out\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907494 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907487    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69cd2ae2-6450-4396-8ab8-97075bf3978f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907766 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907522    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907766 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907551    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktw4s\" (UniqueName: \"kubernetes.io/projected/69cd2ae2-6450-4396-8ab8-97075bf3978f-kube-api-access-ktw4s\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907766 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907571    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/69cd2ae2-6450-4396-8ab8-97075bf3978f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907766 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907595    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907766 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907611    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907766 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907627    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69cd2ae2-6450-4396-8ab8-97075bf3978f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.907766 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.907642    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-web-config\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.908997 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.908970    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69cd2ae2-6450-4396-8ab8-97075bf3978f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.909108 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.909021    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/69cd2ae2-6450-4396-8ab8-97075bf3978f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.910763 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.910174    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69cd2ae2-6450-4396-8ab8-97075bf3978f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.910763 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:33.910181    2522 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 24 21:16:33.910763 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:16:33.910262    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-main-tls podName:69cd2ae2-6450-4396-8ab8-97075bf3978f nodeName:}" failed. No retries permitted until 2026-07-24 21:16:34.410242933 +0000 UTC m=+116.441637934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "69cd2ae2-6450-4396-8ab8-97075bf3978f") : secret "alertmanager-main-tls" not found
Jul 24 21:16:33.916319 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.914738    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/69cd2ae2-6450-4396-8ab8-97075bf3978f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.916319 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.914861    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.916319 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.915098    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.916319 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.915500    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-web-config\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.916319 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.915899    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-config-volume\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.916319 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.916190    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.916319 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.916310    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.916650 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.916517    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/69cd2ae2-6450-4396-8ab8-97075bf3978f-config-out\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:33.931733 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:33.931658    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktw4s\" (UniqueName: \"kubernetes.io/projected/69cd2ae2-6450-4396-8ab8-97075bf3978f-kube-api-access-ktw4s\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:34.016774 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:34.016620    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4zps" event={"ID":"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3","Type":"ContainerStarted","Data":"c9ded99dbc5d8943efb43f789faec9c2e55242097a48de3696ff4aea75db8b5e"}
Jul 24 21:16:34.018159 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:34.018088    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf" event={"ID":"0fc70724-9e58-4b11-9c34-3b9563dfd847","Type":"ContainerStarted","Data":"c4afec9ba459ab7130912525cf9bf382a085c21a359067175077aa8f07858073"}
Jul 24 21:16:34.307900 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:34.307487    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:34.313356 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:34.312947    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:34.316550 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:34.316499    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a96bf67-2b22-4d4d-8486-3f01d422d89b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-drrpz\" (UID: \"0a96bf67-2b22-4d4d-8486-3f01d422d89b\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:34.414449 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:34.414409    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:34.417885 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:34.417833    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/69cd2ae2-6450-4396-8ab8-97075bf3978f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"69cd2ae2-6450-4396-8ab8-97075bf3978f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:34.452614 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:34.452582    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"
Jul 24 21:16:34.612916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:34.612846    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:35.104844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:35.104423    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz"]
Jul 24 21:16:35.112643 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:35.112605    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a96bf67_2b22_4d4d_8486_3f01d422d89b.slice/crio-897116a41b3c9928ef1a364f1b20afb0c7179c55e8c19050c4b3d5fd4d4feabd WatchSource:0}: Error finding container 897116a41b3c9928ef1a364f1b20afb0c7179c55e8c19050c4b3d5fd4d4feabd: Status 404 returned error can't find the container with id 897116a41b3c9928ef1a364f1b20afb0c7179c55e8c19050c4b3d5fd4d4feabd
Jul 24 21:16:35.137745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:35.137574    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:16:35.148863 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:35.148351    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cd2ae2_6450_4396_8ab8_97075bf3978f.slice/crio-07f79984ddb96d64718571f8c2d1f83246649e201b915727022de633133b54ef WatchSource:0}: Error finding container 07f79984ddb96d64718571f8c2d1f83246649e201b915727022de633133b54ef: Status 404 returned error can't find the container with id 07f79984ddb96d64718571f8c2d1f83246649e201b915727022de633133b54ef
Jul 24 21:16:35.605367 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:35.605274    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:35.706418 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:35.706378    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:35.922154 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:35.922076    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:36.017628 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.017593    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:36.036711 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.036464    2522 generic.go:356] "Generic (PLEG): container finished" podID="e5ea6b64-385e-40a4-84fb-bc4666e8c2a3" containerID="e80fc4de91dc8f46ddd3016285dc8c998aab0ab4832a00915c6f1562c3f2e142" exitCode=0
Jul 24 21:16:36.037759 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.037431    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4zps" event={"ID":"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3","Type":"ContainerDied","Data":"e80fc4de91dc8f46ddd3016285dc8c998aab0ab4832a00915c6f1562c3f2e142"}
Jul 24 21:16:36.040293 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.040264    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz" event={"ID":"0a96bf67-2b22-4d4d-8486-3f01d422d89b","Type":"ContainerStarted","Data":"abd4e7d38bfbb4eff72f305b9f04ab1e646bd19bb4b48bcc24156014a631cdac"}
Jul 24 21:16:36.040391 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.040327    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz" event={"ID":"0a96bf67-2b22-4d4d-8486-3f01d422d89b","Type":"ContainerStarted","Data":"c477c5c6c39b4541154966bf049241076349a45b3fda4de5182b7b71ae98945d"}
Jul 24 21:16:36.040391 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.040381    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz" event={"ID":"0a96bf67-2b22-4d4d-8486-3f01d422d89b","Type":"ContainerStarted","Data":"897116a41b3c9928ef1a364f1b20afb0c7179c55e8c19050c4b3d5fd4d4feabd"}
Jul 24 21:16:36.042872 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.042764    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf" event={"ID":"0fc70724-9e58-4b11-9c34-3b9563dfd847","Type":"ContainerStarted","Data":"bfe2e2515879870ec537bdad2981c3dd1ae767b5f0c0e9abfac75038214906e6"}
Jul 24 21:16:36.042872 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.042821    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf" event={"ID":"0fc70724-9e58-4b11-9c34-3b9563dfd847","Type":"ContainerStarted","Data":"6ae2aade108d2b32f6dd1c68c8e455b702bf5a9cf3016692d8e73b857964866c"}
Jul 24 21:16:36.042872 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.042841    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf" event={"ID":"0fc70724-9e58-4b11-9c34-3b9563dfd847","Type":"ContainerStarted","Data":"1cd3490603ef4319aea20fb639841138d8ae3eac2dcc3b77e8f095a7f30f3beb"}
Jul 24 21:16:36.044069 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.044047    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"69cd2ae2-6450-4396-8ab8-97075bf3978f","Type":"ContainerStarted","Data":"07f79984ddb96d64718571f8c2d1f83246649e201b915727022de633133b54ef"}
Jul 24 21:16:36.074083 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.073243    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-sznnf" podStartSLOduration=2.285440308 podStartE2EDuration="4.073223778s" totalImagesPullingTime="1.78778347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:36.073053027 +0000 UTC m=+118.104448041" watchObservedRunningTime="2026-07-24 21:16:36.073223778 +0000 UTC m=+118.104618803"
Jul 24 21:16:36.463273 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.463234    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:36.563404 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:36.563371    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:37.280702 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:37.280645    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:37.350144 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:37.349998    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:37.446179 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:37.446147    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:37.506453 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:37.506383    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:38.053848 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.053821    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4zps" event={"ID":"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3","Type":"ContainerStarted","Data":"08c19b4c2a0b7a66aaf7f3ea141f6337b5a2980d74ad99e581f3e9b564c27654"}
Jul 24 21:16:38.053988 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.053880    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4zps" event={"ID":"e5ea6b64-385e-40a4-84fb-bc4666e8c2a3","Type":"ContainerStarted","Data":"14d6a1ea15460483a5f71a4a66d236bf02647fb2738b9abbf1e1f34b1cc217cb"}
Jul 24 21:16:38.055974 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.055949    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz" event={"ID":"0a96bf67-2b22-4d4d-8486-3f01d422d89b","Type":"ContainerStarted","Data":"d4910da0b325a38e3d9dc2fcaf03a552be1f2595b33e400ea9183502fe19050a"}
Jul 24 21:16:38.057915 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.057895    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sqv7q" event={"ID":"043caca9-caf7-46cb-9297-b7be1f5fd9a9","Type":"ContainerStarted","Data":"bcb6af6a9e20790a96b31c4228b9c0e107967d167a65112194a3c20fc17d3f65"}
Jul 24 21:16:38.059518 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.059498    2522 generic.go:356] "Generic (PLEG): container finished" podID="69cd2ae2-6450-4396-8ab8-97075bf3978f" containerID="02ef67339f9a580e76d5ceea2e8bfd6ebdba884f38b1f49f885f6a24c778b9d6" exitCode=0
Jul 24 21:16:38.059595 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.059531    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"69cd2ae2-6450-4396-8ab8-97075bf3978f","Type":"ContainerDied","Data":"02ef67339f9a580e76d5ceea2e8bfd6ebdba884f38b1f49f885f6a24c778b9d6"}
Jul 24 21:16:38.073954 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.073913    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b4zps" podStartSLOduration=4.624077526 podStartE2EDuration="6.073900423s" totalImagesPullingTime="1.333416432s" totalInitContainerRuntime="116.406465ms" isStatefulPod=true podCreationTimestamp="2026-07-24 21:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:38.071800906 +0000 UTC m=+120.103195931" watchObservedRunningTime="2026-07-24 21:16:38.073900423 +0000 UTC m=+120.105295432"
Jul 24 21:16:38.089016 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.088982    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-drrpz" podStartSLOduration=3.4626711070000002 podStartE2EDuration="6.088966909s" totalImagesPullingTime="2.626295802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:38.087952352 +0000 UTC m=+120.119347385" watchObservedRunningTime="2026-07-24 21:16:38.088966909 +0000 UTC m=+120.120361919"
Jul 24 21:16:38.105828 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.105773    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sqv7q" podStartSLOduration=34.381296527 podStartE2EDuration="47.105748193s" totalImagesPullingTime="12.724451666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:38.105093667 +0000 UTC m=+120.136488676" watchObservedRunningTime="2026-07-24 21:16:38.105748193 +0000 UTC m=+120.137143200"
Jul 24 21:16:38.901623 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.901584    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:38.968874 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.968736    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:38.968874 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.968798    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:38.976027 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:38.976001    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:39.066954 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:39.066924    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:16:39.112134 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:39.112106    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55db5f7fb-kxzsx"]
Jul 24 21:16:39.919207 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:39.919187    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:16:40.070420 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:40.070376    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"69cd2ae2-6450-4396-8ab8-97075bf3978f","Type":"ContainerStarted","Data":"a06d61a4f477f51429703bbb39c544c90b4052247bb42443feff06a1e9eb4ef3"}
Jul 24 21:16:40.070528 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:40.070479    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"69cd2ae2-6450-4396-8ab8-97075bf3978f","Type":"ContainerStarted","Data":"a9decb0dd2f2d8e09c27c4b8f60f55f20ad38f9ebb6f66142b73ed16909260db"}
Jul 24 21:16:40.070601 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:40.070532    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"69cd2ae2-6450-4396-8ab8-97075bf3978f","Type":"ContainerStarted","Data":"b983c0448137097d0ee388496d017e1210c4bf84adbb99f2f1c612739c45eac2"}
Jul 24 21:16:40.070658 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:40.070623    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"69cd2ae2-6450-4396-8ab8-97075bf3978f","Type":"ContainerStarted","Data":"b17eb66a6e08aaa88bd8d1eb1d934ea7321e1fb59f65a23a9a213b3596a3082d"}
Jul 24 21:16:40.070739 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:40.070667    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"69cd2ae2-6450-4396-8ab8-97075bf3978f","Type":"ContainerStarted","Data":"850a723016a0c2cc9ca3d145c7bdec7addd61d2109c77463ce6d9531d0dd2943"}
Jul 24 21:16:40.222055 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:40.222032    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:40.312346 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:40.312281    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:40.465757 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:40.465727    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67c49bbfb6-9t74k"]
Jul 24 21:16:40.688076 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:40.688049    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:40.784613 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:40.784581    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:40.885841 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:40.885808    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:41.078252 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:41.078223    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"69cd2ae2-6450-4396-8ab8-97075bf3978f","Type":"ContainerStarted","Data":"7c50377d49867dfc4c901289261caf3d7619704df88cbf19b2bde9831f5d606a"}
Jul 24 21:16:41.104200 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:41.104153    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.679399698 podStartE2EDuration="8.104138011s" totalImagesPullingTime="5.410178144s" totalInitContainerRuntime="14.560169ms" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:41.10183028 +0000 UTC m=+123.133225292" watchObservedRunningTime="2026-07-24 21:16:41.104138011 +0000 UTC m=+123.135533049"
Jul 24 21:16:42.065866 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:42.065829    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:42.124900 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:42.124866    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:47.732492 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:47.732459    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:47.795966 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:47.795939    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:48.229794 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:48.229763    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:16:48.232293 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:48.232276    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9-metrics-certs\") pod \"network-metrics-daemon-krmvj\" (UID: \"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9\") " pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:16:48.329252 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:48.329227    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zlbx8\""
Jul 24 21:16:48.336513 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:48.336490    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krmvj"
Jul 24 21:16:48.487408 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:48.487323    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-krmvj"]
Jul 24 21:16:48.490134 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:48.490107    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc4cf5b_9c84_4c3d_8009_edcb10e69ca9.slice/crio-dea698f165e6ed0512eafbecea2e17ba9b888a65c902d63ca0da76b1ff7a29cc WatchSource:0}: Error finding container dea698f165e6ed0512eafbecea2e17ba9b888a65c902d63ca0da76b1ff7a29cc: Status 404 returned error can't find the container with id dea698f165e6ed0512eafbecea2e17ba9b888a65c902d63ca0da76b1ff7a29cc
Jul 24 21:16:49.084839 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:49.084773    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:49.115005 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:49.114927    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krmvj" event={"ID":"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9","Type":"ContainerStarted","Data":"dea698f165e6ed0512eafbecea2e17ba9b888a65c902d63ca0da76b1ff7a29cc"}
Jul 24 21:16:49.144880 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:49.144840    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:50.121624 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:50.121586    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krmvj" event={"ID":"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9","Type":"ContainerStarted","Data":"e30c5782dcd1ab422ae6a3d3f8e17812b62866665ff68dfdf44d784aecde0e3d"}
Jul 24 21:16:50.122088 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:50.121650    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krmvj" event={"ID":"fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9","Type":"ContainerStarted","Data":"78d234ffda1747177c348ebb69d8167098107aafa56bbd81ea9a25327c0adbaf"}
Jul 24 21:16:50.154847 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:50.154788    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-krmvj" podStartSLOduration=130.21471488 podStartE2EDuration="2m11.154768281s" totalImagesPullingTime="940.053401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:50.151493846 +0000 UTC m=+132.182888855" watchObservedRunningTime="2026-07-24 21:16:50.154768281 +0000 UTC m=+132.186163290"
Jul 24 21:16:50.184895 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:50.184859    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:50.285174 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:50.285144    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:50.385024 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:50.384946    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:54.646881 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.646852    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b74d5945d-xbnk8"]
Jul 24 21:16:54.649813 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.649793    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.669249 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.669224    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b74d5945d-xbnk8"]
Jul 24 21:16:54.780962 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.780931    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-trusted-ca-bundle\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.781127 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.780976    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-serving-cert\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.781127 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.780999    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-oauth-config\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.781127 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.781030    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-console-config\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.781127 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.781059    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-oauth-serving-cert\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.781256 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.781153    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-service-ca\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.781256 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.781184    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c545d\" (UniqueName: \"kubernetes.io/projected/e0971176-e101-4544-a4fa-43112027dfc8-kube-api-access-c545d\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.794221 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.794189    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:54.881972 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.881947    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-serving-cert\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.882125 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.881993    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-oauth-config\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.882125 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.882017    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-console-config\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.882125 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.882050    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-oauth-serving-cert\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.882306 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.882130    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-service-ca\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.882306 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.882154    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c545d\" (UniqueName: \"kubernetes.io/projected/e0971176-e101-4544-a4fa-43112027dfc8-kube-api-access-c545d\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.882306 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.882191    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-trusted-ca-bundle\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.882960 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.882900    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-console-config\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.883078 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.882958    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-service-ca\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.883143 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.883073    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-trusted-ca-bundle\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.883143 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.883113    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-oauth-serving-cert\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.884570 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.884550    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-oauth-config\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.884734 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.884719    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-serving-cert\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.890849 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.890828    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c545d\" (UniqueName: \"kubernetes.io/projected/e0971176-e101-4544-a4fa-43112027dfc8-kube-api-access-c545d\") pod \"console-7b74d5945d-xbnk8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") " pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:54.900008 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.899943    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:54.958370 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:54.958326    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:16:55.114734 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:55.114709    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b74d5945d-xbnk8"]
Jul 24 21:16:55.118193 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:16:55.118138    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0971176_e101_4544_a4fa_43112027dfc8.slice/crio-8b46b7d70b3f4a391b62d5b1cf857abc0e7dc8826a4490bb06a6f23f6e38df03 WatchSource:0}: Error finding container 8b46b7d70b3f4a391b62d5b1cf857abc0e7dc8826a4490bb06a6f23f6e38df03: Status 404 returned error can't find the container with id 8b46b7d70b3f4a391b62d5b1cf857abc0e7dc8826a4490bb06a6f23f6e38df03
Jul 24 21:16:55.143933 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:55.143902    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b74d5945d-xbnk8" event={"ID":"e0971176-e101-4544-a4fa-43112027dfc8","Type":"ContainerStarted","Data":"8b46b7d70b3f4a391b62d5b1cf857abc0e7dc8826a4490bb06a6f23f6e38df03"}
Jul 24 21:16:55.403425 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:55.403393    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:55.485434 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:55.485408    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:56.149503 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:56.149464    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b74d5945d-xbnk8" event={"ID":"e0971176-e101-4544-a4fa-43112027dfc8","Type":"ContainerStarted","Data":"37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d"}
Jul 24 21:16:56.174105 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:56.174050    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b74d5945d-xbnk8" podStartSLOduration=2.17402892 podStartE2EDuration="2.17402892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:56.173466484 +0000 UTC m=+138.204861493" watchObservedRunningTime="2026-07-24 21:16:56.17402892 +0000 UTC m=+138.205423930"
Jul 24 21:16:56.737769 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:56.737732    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:56.838912 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:16:56.838881    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:02.748958 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:02.748922    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:02.846298 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:02.846260    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:03.178884 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:03.178812    2522 generic.go:356] "Generic (PLEG): container finished" podID="96f53b85-bcf8-4a28-9a05-5738e0e9e873" containerID="5a58934723d4bb618fc708efdf164563282cc5d7c5b953731446b2f0c12c6e93" exitCode=0
Jul 24 21:17:03.179006 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:03.178881    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9" event={"ID":"96f53b85-bcf8-4a28-9a05-5738e0e9e873","Type":"ContainerDied","Data":"5a58934723d4bb618fc708efdf164563282cc5d7c5b953731446b2f0c12c6e93"}
Jul 24 21:17:03.179185 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:03.179169    2522 scope.go:117] "RemoveContainer" containerID="5a58934723d4bb618fc708efdf164563282cc5d7c5b953731446b2f0c12c6e93"
Jul 24 21:17:04.136808 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.136763    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55db5f7fb-kxzsx" podUID="061e4c22-f3b9-4b63-a7ec-78d2154fc989" containerName="console" containerID="cri-o://6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760" gracePeriod=15
Jul 24 21:17:04.165379 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:17:04.165351    2522 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061e4c22_f3b9_4b63_a7ec_78d2154fc989.slice/crio-6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760.scope\": RecentStats: unable to find data in memory cache]"
Jul 24 21:17:04.186701 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.186656    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-llmz9" event={"ID":"96f53b85-bcf8-4a28-9a05-5738e0e9e873","Type":"ContainerStarted","Data":"da932f6e2d31c3e032d63108055a158a6d29513614dbb7246c905c81a6ebb7a9"}
Jul 24 21:17:04.447031 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.447001    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:17:04.553695 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.553659    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-oauth-serving-cert\") pod \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") "
Jul 24 21:17:04.553836 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.553700    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djkbk\" (UniqueName: \"kubernetes.io/projected/061e4c22-f3b9-4b63-a7ec-78d2154fc989-kube-api-access-djkbk\") pod \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") "
Jul 24 21:17:04.553836 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.553754    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-config\") pod \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") "
Jul 24 21:17:04.553836 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.553786    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-service-ca\") pod \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") "
Jul 24 21:17:04.553836 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.553819    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-serving-cert\") pod \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") "
Jul 24 21:17:04.554040 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.553918    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-oauth-config\") pod \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\" (UID: \"061e4c22-f3b9-4b63-a7ec-78d2154fc989\") "
Jul 24 21:17:04.554126 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.554100    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "061e4c22-f3b9-4b63-a7ec-78d2154fc989" (UID: "061e4c22-f3b9-4b63-a7ec-78d2154fc989"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:04.554186 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.554139    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-service-ca" (OuterVolumeSpecName: "service-ca") pod "061e4c22-f3b9-4b63-a7ec-78d2154fc989" (UID: "061e4c22-f3b9-4b63-a7ec-78d2154fc989"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:04.554186 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.554155    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-config" (OuterVolumeSpecName: "console-config") pod "061e4c22-f3b9-4b63-a7ec-78d2154fc989" (UID: "061e4c22-f3b9-4b63-a7ec-78d2154fc989"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:04.554267 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.554234    2522 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-oauth-serving-cert\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:04.554267 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.554245    2522 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:04.554267 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.554257    2522 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/061e4c22-f3b9-4b63-a7ec-78d2154fc989-service-ca\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:04.556245 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.556224    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "061e4c22-f3b9-4b63-a7ec-78d2154fc989" (UID: "061e4c22-f3b9-4b63-a7ec-78d2154fc989"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:17:04.556433 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.556405    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "061e4c22-f3b9-4b63-a7ec-78d2154fc989" (UID: "061e4c22-f3b9-4b63-a7ec-78d2154fc989"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:17:04.556433 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.556426    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061e4c22-f3b9-4b63-a7ec-78d2154fc989-kube-api-access-djkbk" (OuterVolumeSpecName: "kube-api-access-djkbk") pod "061e4c22-f3b9-4b63-a7ec-78d2154fc989" (UID: "061e4c22-f3b9-4b63-a7ec-78d2154fc989"). InnerVolumeSpecName "kube-api-access-djkbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:17:04.655367 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.655299    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djkbk\" (UniqueName: \"kubernetes.io/projected/061e4c22-f3b9-4b63-a7ec-78d2154fc989-kube-api-access-djkbk\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:04.655367 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.655323    2522 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-serving-cert\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:04.655367 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.655335    2522 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/061e4c22-f3b9-4b63-a7ec-78d2154fc989-console-oauth-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:04.959388 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.959363    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:17:04.959503 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.959425    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:17:04.964070 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:04.964051    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:17:05.192596 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.192571    2522 generic.go:356] "Generic (PLEG): container finished" podID="061e4c22-f3b9-4b63-a7ec-78d2154fc989" containerID="6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760" exitCode=0
Jul 24 21:17:05.192949 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.192617    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55db5f7fb-kxzsx"
Jul 24 21:17:05.192949 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.192662    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55db5f7fb-kxzsx" event={"ID":"061e4c22-f3b9-4b63-a7ec-78d2154fc989","Type":"ContainerDied","Data":"6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760"}
Jul 24 21:17:05.192949 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.192734    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55db5f7fb-kxzsx" event={"ID":"061e4c22-f3b9-4b63-a7ec-78d2154fc989","Type":"ContainerDied","Data":"eb15470a375561950f253589546abcda2622c0a83c2b8f3bf673ba87545a695b"}
Jul 24 21:17:05.192949 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.192756    2522 scope.go:117] "RemoveContainer" containerID="6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760"
Jul 24 21:17:05.203167 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.203143    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:17:05.206960 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.206945    2522 scope.go:117] "RemoveContainer" containerID="6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760"
Jul 24 21:17:05.207281 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:17:05.207261    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760\": container with ID starting with 6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760 not found: ID does not exist" containerID="6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760"
Jul 24 21:17:05.207355 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.207294    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760"} err="failed to get container status \"6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760\": rpc error: code = NotFound desc = could not find container \"6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760\": container with ID starting with 6686f8a1ead9e7844799bb760ae178dc1aa13b16ccd048b1e6e9b516093cb760 not found: ID does not exist"
Jul 24 21:17:05.247799 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.247752    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55db5f7fb-kxzsx"]
Jul 24 21:17:05.249972 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.249950    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55db5f7fb-kxzsx"]
Jul 24 21:17:05.262545 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.262520    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7489dc58b6-4q5tb"]
Jul 24 21:17:05.373470 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.373439    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:05.477968 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.477925    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:05.484451 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.484421    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k" podUID="104d73c0-ff20-47b8-9774-6b3eaa2708ae" containerName="registry" containerID="cri-o://c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71" gracePeriod=30
Jul 24 21:17:05.539984 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.539955    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:05.760158 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.760136    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:17:05.864640 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.864581    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/104d73c0-ff20-47b8-9774-6b3eaa2708ae-ca-trust-extracted\") pod \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") "
Jul 24 21:17:05.864640 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.864608    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-bound-sa-token\") pod \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") "
Jul 24 21:17:05.864640 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.864637    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b257t\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-kube-api-access-b257t\") pod \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") "
Jul 24 21:17:05.864899 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.864691    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-trusted-ca\") pod \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") "
Jul 24 21:17:05.864899 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.864736    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-image-registry-private-configuration\") pod \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") "
Jul 24 21:17:05.864899 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.864773    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-installation-pull-secrets\") pod \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") "
Jul 24 21:17:05.864899 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.864805    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") pod \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") "
Jul 24 21:17:05.865172 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.865144    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-certificates\") pod \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\" (UID: \"104d73c0-ff20-47b8-9774-6b3eaa2708ae\") "
Jul 24 21:17:05.865286 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.865204    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "104d73c0-ff20-47b8-9774-6b3eaa2708ae" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:05.865602 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.865580    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "104d73c0-ff20-47b8-9774-6b3eaa2708ae" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:05.865602 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.865589    2522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-trusted-ca\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:05.867873 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.867830    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "104d73c0-ff20-47b8-9774-6b3eaa2708ae" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:17:05.867873 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.867870    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "104d73c0-ff20-47b8-9774-6b3eaa2708ae" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:17:05.868024 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.867911    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "104d73c0-ff20-47b8-9774-6b3eaa2708ae" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:17:05.868024 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.867993    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "104d73c0-ff20-47b8-9774-6b3eaa2708ae" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:17:05.868024 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.867993    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-kube-api-access-b257t" (OuterVolumeSpecName: "kube-api-access-b257t") pod "104d73c0-ff20-47b8-9774-6b3eaa2708ae" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae"). InnerVolumeSpecName "kube-api-access-b257t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:17:05.874359 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.874330    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104d73c0-ff20-47b8-9774-6b3eaa2708ae-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "104d73c0-ff20-47b8-9774-6b3eaa2708ae" (UID: "104d73c0-ff20-47b8-9774-6b3eaa2708ae"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:17:05.966418 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.966397    2522 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/104d73c0-ff20-47b8-9774-6b3eaa2708ae-ca-trust-extracted\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:05.966418 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.966417    2522 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-bound-sa-token\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:05.966529 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.966428    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b257t\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-kube-api-access-b257t\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:05.966529 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.966438    2522 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-image-registry-private-configuration\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:05.966529 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.966447    2522 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/104d73c0-ff20-47b8-9774-6b3eaa2708ae-installation-pull-secrets\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:05.966529 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.966455    2522 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:05.966529 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:05.966464    2522 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/104d73c0-ff20-47b8-9774-6b3eaa2708ae-registry-certificates\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:06.198739 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:06.198713    2522 generic.go:356] "Generic (PLEG): container finished" podID="104d73c0-ff20-47b8-9774-6b3eaa2708ae" containerID="c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71" exitCode=0
Jul 24 21:17:06.199183 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:06.198767    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k"
Jul 24 21:17:06.199183 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:06.198806    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k" event={"ID":"104d73c0-ff20-47b8-9774-6b3eaa2708ae","Type":"ContainerDied","Data":"c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71"}
Jul 24 21:17:06.199183 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:06.198863    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67c49bbfb6-9t74k" event={"ID":"104d73c0-ff20-47b8-9774-6b3eaa2708ae","Type":"ContainerDied","Data":"0c96e3f7678bbcc59cacaf969454504b83e629db578c819b597d4a7e0e0777d4"}
Jul 24 21:17:06.199183 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:06.198885    2522 scope.go:117] "RemoveContainer" containerID="c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71"
Jul 24 21:17:06.207832 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:06.207655    2522 scope.go:117] "RemoveContainer" containerID="c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71"
Jul 24 21:17:06.208037 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:17:06.208020    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71\": container with ID starting with c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71 not found: ID does not exist" containerID="c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71"
Jul 24 21:17:06.208091 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:06.208046    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71"} err="failed to get container status \"c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71\": rpc error: code = NotFound desc = could not find container \"c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71\": container with ID starting with c11d6492281e163a5ad9986b8fdc7585b293557e3fdfb57f671fc3dcdd6aaa71 not found: ID does not exist"
Jul 24 21:17:06.220199 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:06.220172    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67c49bbfb6-9t74k"]
Jul 24 21:17:06.224341 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:06.224317    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67c49bbfb6-9t74k"]
Jul 24 21:17:06.405974 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:06.405946    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="061e4c22-f3b9-4b63-a7ec-78d2154fc989" path="/var/lib/kubelet/pods/061e4c22-f3b9-4b63-a7ec-78d2154fc989/volumes"
Jul 24 21:17:06.406324 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:06.406311    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104d73c0-ff20-47b8-9774-6b3eaa2708ae" path="/var/lib/kubelet/pods/104d73c0-ff20-47b8-9774-6b3eaa2708ae/volumes"
Jul 24 21:17:16.502562 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:16.502529    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:17.733168 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:17.733114    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:17.803365 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:17.803333    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:30.287001 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.286860    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7489dc58b6-4q5tb" podUID="20d225c2-33d2-4f52-ab38-27848ffc551f" containerName="console" containerID="cri-o://4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd" gracePeriod=15
Jul 24 21:17:30.534186 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.534164    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:17:30.548851 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.548804    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-oauth-serving-cert\") pod \"20d225c2-33d2-4f52-ab38-27848ffc551f\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") "
Jul 24 21:17:30.548851 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.548836    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5xg2\" (UniqueName: \"kubernetes.io/projected/20d225c2-33d2-4f52-ab38-27848ffc551f-kube-api-access-h5xg2\") pod \"20d225c2-33d2-4f52-ab38-27848ffc551f\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") "
Jul 24 21:17:30.548959 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.548866    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-serving-cert\") pod \"20d225c2-33d2-4f52-ab38-27848ffc551f\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") "
Jul 24 21:17:30.548959 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.548896    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-trusted-ca-bundle\") pod \"20d225c2-33d2-4f52-ab38-27848ffc551f\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") "
Jul 24 21:17:30.548959 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.548915    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-console-config\") pod \"20d225c2-33d2-4f52-ab38-27848ffc551f\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") "
Jul 24 21:17:30.548959 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.548947    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-oauth-config\") pod \"20d225c2-33d2-4f52-ab38-27848ffc551f\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") "
Jul 24 21:17:30.549146 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.548987    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-service-ca\") pod \"20d225c2-33d2-4f52-ab38-27848ffc551f\" (UID: \"20d225c2-33d2-4f52-ab38-27848ffc551f\") "
Jul 24 21:17:30.549310 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.549276    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "20d225c2-33d2-4f52-ab38-27848ffc551f" (UID: "20d225c2-33d2-4f52-ab38-27848ffc551f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:30.549383 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.549346    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "20d225c2-33d2-4f52-ab38-27848ffc551f" (UID: "20d225c2-33d2-4f52-ab38-27848ffc551f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:30.549433 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.549382    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-console-config" (OuterVolumeSpecName: "console-config") pod "20d225c2-33d2-4f52-ab38-27848ffc551f" (UID: "20d225c2-33d2-4f52-ab38-27848ffc551f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:30.549469 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.549445    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-service-ca" (OuterVolumeSpecName: "service-ca") pod "20d225c2-33d2-4f52-ab38-27848ffc551f" (UID: "20d225c2-33d2-4f52-ab38-27848ffc551f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:30.551051 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.551018    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d225c2-33d2-4f52-ab38-27848ffc551f-kube-api-access-h5xg2" (OuterVolumeSpecName: "kube-api-access-h5xg2") pod "20d225c2-33d2-4f52-ab38-27848ffc551f" (UID: "20d225c2-33d2-4f52-ab38-27848ffc551f"). InnerVolumeSpecName "kube-api-access-h5xg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:17:30.551597 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.551572    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "20d225c2-33d2-4f52-ab38-27848ffc551f" (UID: "20d225c2-33d2-4f52-ab38-27848ffc551f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:17:30.551597 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.551582    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "20d225c2-33d2-4f52-ab38-27848ffc551f" (UID: "20d225c2-33d2-4f52-ab38-27848ffc551f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:17:30.650324 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.650296    2522 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-serving-cert\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:30.650324 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.650319    2522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-trusted-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:30.650324 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.650329    2522 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-console-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:30.650521 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.650338    2522 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20d225c2-33d2-4f52-ab38-27848ffc551f-console-oauth-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:30.650521 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.650355    2522 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-service-ca\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:30.650521 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.650364    2522 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20d225c2-33d2-4f52-ab38-27848ffc551f-oauth-serving-cert\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:30.650521 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:30.650373    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5xg2\" (UniqueName: \"kubernetes.io/projected/20d225c2-33d2-4f52-ab38-27848ffc551f-kube-api-access-h5xg2\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:17:31.310078 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:31.310038    2522 generic.go:356] "Generic (PLEG): container finished" podID="20d225c2-33d2-4f52-ab38-27848ffc551f" containerID="4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd" exitCode=0
Jul 24 21:17:31.310467 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:31.310090    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7489dc58b6-4q5tb"
Jul 24 21:17:31.310467 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:31.310124    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7489dc58b6-4q5tb" event={"ID":"20d225c2-33d2-4f52-ab38-27848ffc551f","Type":"ContainerDied","Data":"4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd"}
Jul 24 21:17:31.310467 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:31.310184    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7489dc58b6-4q5tb" event={"ID":"20d225c2-33d2-4f52-ab38-27848ffc551f","Type":"ContainerDied","Data":"f5c276d2f201b7c0648ecab11818a3be12c9849c13ea6d233466a2ecc76c4d65"}
Jul 24 21:17:31.310467 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:31.310207    2522 scope.go:117] "RemoveContainer" containerID="4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd"
Jul 24 21:17:31.319730 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:31.319710    2522 scope.go:117] "RemoveContainer" containerID="4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd"
Jul 24 21:17:31.320013 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:17:31.319991    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd\": container with ID starting with 4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd not found: ID does not exist" containerID="4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd"
Jul 24 21:17:31.320058 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:31.320024    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd"} err="failed to get container status \"4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd\": rpc error: code = NotFound desc = could not find container \"4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd\": container with ID starting with 4d61297096835680ab0aa3612a78cd2f752877fb533ef6ae577dfedbbabbcdcd not found: ID does not exist"
Jul 24 21:17:31.331307 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:31.331281    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7489dc58b6-4q5tb"]
Jul 24 21:17:31.335049 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:31.335025    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7489dc58b6-4q5tb"]
Jul 24 21:17:32.406599 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:32.406559    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20d225c2-33d2-4f52-ab38-27848ffc551f" path="/var/lib/kubelet/pods/20d225c2-33d2-4f52-ab38-27848ffc551f/volumes"
Jul 24 21:17:48.770941 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:48.770909    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:48.824933 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:48.824896    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:52.931264 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.931224    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-548f886447-zxwfc"]
Jul 24 21:17:52.931644 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.931600    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104d73c0-ff20-47b8-9774-6b3eaa2708ae" containerName="registry"
Jul 24 21:17:52.931644 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.931615    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="104d73c0-ff20-47b8-9774-6b3eaa2708ae" containerName="registry"
Jul 24 21:17:52.931644 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.931634    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061e4c22-f3b9-4b63-a7ec-78d2154fc989" containerName="console"
Jul 24 21:17:52.931644 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.931639    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="061e4c22-f3b9-4b63-a7ec-78d2154fc989" containerName="console"
Jul 24 21:17:52.931839 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.931646    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d225c2-33d2-4f52-ab38-27848ffc551f" containerName="console"
Jul 24 21:17:52.931839 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.931655    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d225c2-33d2-4f52-ab38-27848ffc551f" containerName="console"
Jul 24 21:17:52.931839 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.931781    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="104d73c0-ff20-47b8-9774-6b3eaa2708ae" containerName="registry"
Jul 24 21:17:52.931839 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.931795    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="061e4c22-f3b9-4b63-a7ec-78d2154fc989" containerName="console"
Jul 24 21:17:52.931839 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.931801    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d225c2-33d2-4f52-ab38-27848ffc551f" containerName="console"
Jul 24 21:17:52.936580 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.936559    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:52.948131 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.948106    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-548f886447-zxwfc"]
Jul 24 21:17:52.994094 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:52.994066    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:53.008427 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.008402    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-trusted-ca-bundle\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.008524 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.008435    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-oauth-serving-cert\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.008524 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.008481    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-serving-cert\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.008524 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.008504    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxpq\" (UniqueName: \"kubernetes.io/projected/ea7c8b57-df75-4cbc-a4f2-6734db324d84-kube-api-access-szxpq\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.008524 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.008523    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-oauth-config\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.008710 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.008539    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-config\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.008710 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.008592    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-service-ca\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.047054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.047025    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:53.109769 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.109737    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-oauth-serving-cert\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.109884 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.109787    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-serving-cert\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.109884 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.109809    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxpq\" (UniqueName: \"kubernetes.io/projected/ea7c8b57-df75-4cbc-a4f2-6734db324d84-kube-api-access-szxpq\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.109884 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.109828    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-oauth-config\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.109884 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.109843    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-config\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.109884 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.109864    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-service-ca\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.110111 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.109901    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-trusted-ca-bundle\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.110562 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.110530    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-oauth-serving-cert\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.110659 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.110530    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-service-ca\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.110659 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.110614    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-config\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.110826 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.110810    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-trusted-ca-bundle\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.112362 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.112342    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-oauth-config\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.112468 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.112450    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-serving-cert\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.118041 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.118013    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxpq\" (UniqueName: \"kubernetes.io/projected/ea7c8b57-df75-4cbc-a4f2-6734db324d84-kube-api-access-szxpq\") pod \"console-548f886447-zxwfc\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") " pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.248790 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.248767    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:17:53.380469 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.380428    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-548f886447-zxwfc"]
Jul 24 21:17:53.383264 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:17:53.383234    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7c8b57_df75_4cbc_a4f2_6734db324d84.slice/crio-9c4c8abcbe7791cb446790a86f18cb3386f2cf10f989834413ec02158d354dcc WatchSource:0}: Error finding container 9c4c8abcbe7791cb446790a86f18cb3386f2cf10f989834413ec02158d354dcc: Status 404 returned error can't find the container with id 9c4c8abcbe7791cb446790a86f18cb3386f2cf10f989834413ec02158d354dcc
Jul 24 21:17:53.399541 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:53.399442    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-548f886447-zxwfc" event={"ID":"ea7c8b57-df75-4cbc-a4f2-6734db324d84","Type":"ContainerStarted","Data":"9c4c8abcbe7791cb446790a86f18cb3386f2cf10f989834413ec02158d354dcc"}
Jul 24 21:17:54.405722 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:54.405664    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-548f886447-zxwfc" event={"ID":"ea7c8b57-df75-4cbc-a4f2-6734db324d84","Type":"ContainerStarted","Data":"8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6"}
Jul 24 21:17:54.422848 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:17:54.422803    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-548f886447-zxwfc" podStartSLOduration=2.422786731 podStartE2EDuration="2.422786731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:17:54.421606167 +0000 UTC m=+196.453001175" watchObservedRunningTime="2026-07-24 21:17:54.422786731 +0000 UTC m=+196.454181738"
Jul 24 21:18:00.734237 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:00.734193    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:00.835358 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:00.835319    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:03.249483 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:03.249440    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:18:03.249942 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:03.249506    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:18:03.254492 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:03.254467    2522 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:18:03.441586 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:03.441556    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:18:03.479298 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:03.479269    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b74d5945d-xbnk8"]
Jul 24 21:18:03.579023 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:03.578946    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:03.680252 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:03.680217    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:03.735049 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:03.735025    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:17.742830 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:17.742789    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:17.839515 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:17.839486    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:28.500746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.500705    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b74d5945d-xbnk8" podUID="e0971176-e101-4544-a4fa-43112027dfc8" containerName="console" containerID="cri-o://37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d" gracePeriod=15
Jul 24 21:18:28.748516 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.748495    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:18:28.870511 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.870455    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-service-ca\") pod \"e0971176-e101-4544-a4fa-43112027dfc8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") "
Jul 24 21:18:28.870511 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.870484    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-console-config\") pod \"e0971176-e101-4544-a4fa-43112027dfc8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") "
Jul 24 21:18:28.870848 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.870516    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-oauth-config\") pod \"e0971176-e101-4544-a4fa-43112027dfc8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") "
Jul 24 21:18:28.870848 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.870533    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-trusted-ca-bundle\") pod \"e0971176-e101-4544-a4fa-43112027dfc8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") "
Jul 24 21:18:28.870848 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.870695    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-serving-cert\") pod \"e0971176-e101-4544-a4fa-43112027dfc8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") "
Jul 24 21:18:28.870848 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.870764    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c545d\" (UniqueName: \"kubernetes.io/projected/e0971176-e101-4544-a4fa-43112027dfc8-kube-api-access-c545d\") pod \"e0971176-e101-4544-a4fa-43112027dfc8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") "
Jul 24 21:18:28.871056 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.870856    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-oauth-serving-cert\") pod \"e0971176-e101-4544-a4fa-43112027dfc8\" (UID: \"e0971176-e101-4544-a4fa-43112027dfc8\") "
Jul 24 21:18:28.871056 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.870926    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-console-config" (OuterVolumeSpecName: "console-config") pod "e0971176-e101-4544-a4fa-43112027dfc8" (UID: "e0971176-e101-4544-a4fa-43112027dfc8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:28.871056 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.871017    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e0971176-e101-4544-a4fa-43112027dfc8" (UID: "e0971176-e101-4544-a4fa-43112027dfc8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:28.871056 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.871028    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-service-ca" (OuterVolumeSpecName: "service-ca") pod "e0971176-e101-4544-a4fa-43112027dfc8" (UID: "e0971176-e101-4544-a4fa-43112027dfc8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:28.871235 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.871212    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e0971176-e101-4544-a4fa-43112027dfc8" (UID: "e0971176-e101-4544-a4fa-43112027dfc8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:28.871235 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.871226    2522 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-service-ca\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:18:28.871343 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.871245    2522 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-console-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:18:28.871343 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.871259    2522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-trusted-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:18:28.873231 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.873197    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e0971176-e101-4544-a4fa-43112027dfc8" (UID: "e0971176-e101-4544-a4fa-43112027dfc8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:28.873397 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.873370    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e0971176-e101-4544-a4fa-43112027dfc8" (UID: "e0971176-e101-4544-a4fa-43112027dfc8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:28.873471 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.873377    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0971176-e101-4544-a4fa-43112027dfc8-kube-api-access-c545d" (OuterVolumeSpecName: "kube-api-access-c545d") pod "e0971176-e101-4544-a4fa-43112027dfc8" (UID: "e0971176-e101-4544-a4fa-43112027dfc8"). InnerVolumeSpecName "kube-api-access-c545d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:18:28.971903 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.971883    2522 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0971176-e101-4544-a4fa-43112027dfc8-oauth-serving-cert\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:18:28.971903 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.971902    2522 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-oauth-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:18:28.972020 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.971911    2522 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0971176-e101-4544-a4fa-43112027dfc8-console-serving-cert\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:18:28.972020 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:28.971919    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c545d\" (UniqueName: \"kubernetes.io/projected/e0971176-e101-4544-a4fa-43112027dfc8-kube-api-access-c545d\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:18:29.543081 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:29.543052    2522 generic.go:356] "Generic (PLEG): container finished" podID="e0971176-e101-4544-a4fa-43112027dfc8" containerID="37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d" exitCode=0
Jul 24 21:18:29.543425 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:29.543110    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b74d5945d-xbnk8"
Jul 24 21:18:29.543425 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:29.543156    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b74d5945d-xbnk8" event={"ID":"e0971176-e101-4544-a4fa-43112027dfc8","Type":"ContainerDied","Data":"37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d"}
Jul 24 21:18:29.543425 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:29.543218    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b74d5945d-xbnk8" event={"ID":"e0971176-e101-4544-a4fa-43112027dfc8","Type":"ContainerDied","Data":"8b46b7d70b3f4a391b62d5b1cf857abc0e7dc8826a4490bb06a6f23f6e38df03"}
Jul 24 21:18:29.543425 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:29.543246    2522 scope.go:117] "RemoveContainer" containerID="37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d"
Jul 24 21:18:29.553318 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:29.553297    2522 scope.go:117] "RemoveContainer" containerID="37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d"
Jul 24 21:18:29.553622 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:18:29.553603    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d\": container with ID starting with 37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d not found: ID does not exist" containerID="37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d"
Jul 24 21:18:29.553666 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:29.553633    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d"} err="failed to get container status \"37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d\": rpc error: code = NotFound desc = could not find container \"37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d\": container with ID starting with 37ed74ecd34b32570f0c55c270dbf368b82d8b720f483f588dabde0ce354753d not found: ID does not exist"
Jul 24 21:18:29.565401 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:29.565376    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b74d5945d-xbnk8"]
Jul 24 21:18:29.570583 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:29.570558    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b74d5945d-xbnk8"]
Jul 24 21:18:30.407051 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:30.407014    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0971176-e101-4544-a4fa-43112027dfc8" path="/var/lib/kubelet/pods/e0971176-e101-4544-a4fa-43112027dfc8/volumes"
Jul 24 21:18:32.609907 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:32.609865    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:32.710139 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:32.710104    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:34.157418 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:18:34.157384    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:06.216687 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:06.216569    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:38.316421 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:38.316391    2522 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 21:19:48.233115 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:48.233076    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:48.302431 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:48.302399    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:48.797278 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:48.797243    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:48.897081 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:48.897055    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:48.961641 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:48.961612    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:49.894856 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.894809    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"]
Jul 24 21:19:49.895382 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.895328    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0971176-e101-4544-a4fa-43112027dfc8" containerName="console"
Jul 24 21:19:49.895382 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.895348    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0971176-e101-4544-a4fa-43112027dfc8" containerName="console"
Jul 24 21:19:49.895501 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.895427    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0971176-e101-4544-a4fa-43112027dfc8" containerName="console"
Jul 24 21:19:49.898518 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.898494    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:19:49.901005 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.900983    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 21:19:49.901775 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.901752    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 21:19:49.901895 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.901796    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nwbcb\""
Jul 24 21:19:49.907542 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.907517    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"]
Jul 24 21:19:49.934089 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.934053    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:19:49.934207 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.934099    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:19:49.934207 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:49.934173    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bhw\" (UniqueName: \"kubernetes.io/projected/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-kube-api-access-p4bhw\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:19:50.035464 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:50.035438    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:19:50.035571 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:50.035471    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:19:50.035571 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:50.035498    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bhw\" (UniqueName: \"kubernetes.io/projected/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-kube-api-access-p4bhw\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:19:50.035879 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:50.035860    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:19:50.035952 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:50.035900    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:19:50.044464 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:50.044439    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bhw\" (UniqueName: \"kubernetes.io/projected/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-kube-api-access-p4bhw\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:19:50.209516 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:50.209494    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:19:50.341702 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:50.341625    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"]
Jul 24 21:19:50.343901 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:19:50.343874    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e974f6_d2bb_48c5_8d50_158e0bfbd0c5.slice/crio-670bd7a67c376d387e2493b01e7c95efeaf6bc132004db3d30d3cca7cf1ddcaa WatchSource:0}: Error finding container 670bd7a67c376d387e2493b01e7c95efeaf6bc132004db3d30d3cca7cf1ddcaa: Status 404 returned error can't find the container with id 670bd7a67c376d387e2493b01e7c95efeaf6bc132004db3d30d3cca7cf1ddcaa
Jul 24 21:19:50.345909 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:50.345892    2522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:19:50.860364 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:50.860316    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd" event={"ID":"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5","Type":"ContainerStarted","Data":"670bd7a67c376d387e2493b01e7c95efeaf6bc132004db3d30d3cca7cf1ddcaa"}
Jul 24 21:19:55.886637 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:55.886607    2522 generic.go:356] "Generic (PLEG): container finished" podID="25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" containerID="d2e65524593d402e17703ebcc1c17d78f7abe206f5237d8107c1c0fb2136ce5b" exitCode=0
Jul 24 21:19:55.887011 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:55.886654    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd" event={"ID":"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5","Type":"ContainerDied","Data":"d2e65524593d402e17703ebcc1c17d78f7abe206f5237d8107c1c0fb2136ce5b"}
Jul 24 21:19:58.904403 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:58.904363    2522 generic.go:356] "Generic (PLEG): container finished" podID="25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" containerID="073d7a48f5f85c3008722da0a73c34cb6e3aa92ccf6bc1daddb12be159ce761b" exitCode=0
Jul 24 21:19:58.904846 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:19:58.904413    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd" event={"ID":"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5","Type":"ContainerDied","Data":"073d7a48f5f85c3008722da0a73c34cb6e3aa92ccf6bc1daddb12be159ce761b"}
Jul 24 21:20:04.932155 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:04.932123    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd" event={"ID":"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5","Type":"ContainerStarted","Data":"8714a942e9182a3c05d97fdd81390bdac9d45c452438fbcc80335f2a6abe6ecd"}
Jul 24 21:20:04.951046 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:04.951002    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd" podStartSLOduration=2.958675978 podStartE2EDuration="15.950984656s" totalImagesPullingTime="12.992308678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:19:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:20:04.949303544 +0000 UTC m=+326.980698552" watchObservedRunningTime="2026-07-24 21:20:04.950984656 +0000 UTC m=+326.982379663"
Jul 24 21:20:05.938988 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:05.938947    2522 generic.go:356] "Generic (PLEG): container finished" podID="25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" containerID="8714a942e9182a3c05d97fdd81390bdac9d45c452438fbcc80335f2a6abe6ecd" exitCode=0
Jul 24 21:20:05.939392 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:05.939010    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd" event={"ID":"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5","Type":"ContainerDied","Data":"8714a942e9182a3c05d97fdd81390bdac9d45c452438fbcc80335f2a6abe6ecd"}
Jul 24 21:20:06.125970 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:06.125932    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:07.076090 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.076067    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:20:07.173409 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.173383    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-util\") pod \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") "
Jul 24 21:20:07.173513 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.173416    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-bundle\") pod \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") "
Jul 24 21:20:07.173513 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.173435    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4bhw\" (UniqueName: \"kubernetes.io/projected/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-kube-api-access-p4bhw\") pod \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\" (UID: \"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5\") "
Jul 24 21:20:07.174067 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.174037    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-bundle" (OuterVolumeSpecName: "bundle") pod "25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" (UID: "25e974f6-d2bb-48c5-8d50-158e0bfbd0c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:20:07.175633 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.175606    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-kube-api-access-p4bhw" (OuterVolumeSpecName: "kube-api-access-p4bhw") pod "25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" (UID: "25e974f6-d2bb-48c5-8d50-158e0bfbd0c5"). InnerVolumeSpecName "kube-api-access-p4bhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:20:07.178175 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.178155    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-util" (OuterVolumeSpecName: "util") pod "25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" (UID: "25e974f6-d2bb-48c5-8d50-158e0bfbd0c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:20:07.274775 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.274715    2522 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-util\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:20:07.274775 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.274743    2522 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:20:07.274775 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.274752    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bhw\" (UniqueName: \"kubernetes.io/projected/25e974f6-d2bb-48c5-8d50-158e0bfbd0c5-kube-api-access-p4bhw\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:20:07.949532 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.949496    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd" event={"ID":"25e974f6-d2bb-48c5-8d50-158e0bfbd0c5","Type":"ContainerDied","Data":"670bd7a67c376d387e2493b01e7c95efeaf6bc132004db3d30d3cca7cf1ddcaa"}
Jul 24 21:20:07.949532 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.949514    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5545wd"
Jul 24 21:20:07.949782 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:07.949551    2522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="670bd7a67c376d387e2493b01e7c95efeaf6bc132004db3d30d3cca7cf1ddcaa"
Jul 24 21:20:11.028638 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:11.028601    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:11.131056 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:11.131013    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:11.573376 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:11.573341    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:11.673654 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:11.673620    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:12.830627 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.830588    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"]
Jul 24 21:20:12.831015 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.830919    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" containerName="util"
Jul 24 21:20:12.831015 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.830932    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" containerName="util"
Jul 24 21:20:12.831015 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.830943    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" containerName="extract"
Jul 24 21:20:12.831015 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.830949    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" containerName="extract"
Jul 24 21:20:12.831015 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.830963    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" containerName="pull"
Jul 24 21:20:12.831015 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.830969    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" containerName="pull"
Jul 24 21:20:12.831226 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.831022    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e974f6-d2bb-48c5-8d50-158e0bfbd0c5" containerName="extract"
Jul 24 21:20:12.882461 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.882431    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"]
Jul 24 21:20:12.882594 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.882533    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"
Jul 24 21:20:12.886748 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.886721    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-lkc84\""
Jul 24 21:20:12.886879 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.886730    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 24 21:20:12.887038 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.887023    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 24 21:20:12.887138 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:12.887118    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 24 21:20:13.013571 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:13.013551    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bchd\" (UniqueName: \"kubernetes.io/projected/9b7c74fa-ae1d-441d-923d-0215f2408105-kube-api-access-2bchd\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-h79bs\" (UID: \"9b7c74fa-ae1d-441d-923d-0215f2408105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"
Jul 24 21:20:13.013695 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:13.013577    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9b7c74fa-ae1d-441d-923d-0215f2408105-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-h79bs\" (UID: \"9b7c74fa-ae1d-441d-923d-0215f2408105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"
Jul 24 21:20:13.114909 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:13.114859    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bchd\" (UniqueName: \"kubernetes.io/projected/9b7c74fa-ae1d-441d-923d-0215f2408105-kube-api-access-2bchd\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-h79bs\" (UID: \"9b7c74fa-ae1d-441d-923d-0215f2408105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"
Jul 24 21:20:13.114909 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:13.114884    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9b7c74fa-ae1d-441d-923d-0215f2408105-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-h79bs\" (UID: \"9b7c74fa-ae1d-441d-923d-0215f2408105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"
Jul 24 21:20:13.117199 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:13.117183    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/9b7c74fa-ae1d-441d-923d-0215f2408105-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-h79bs\" (UID: \"9b7c74fa-ae1d-441d-923d-0215f2408105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"
Jul 24 21:20:13.123559 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:13.123540    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bchd\" (UniqueName: \"kubernetes.io/projected/9b7c74fa-ae1d-441d-923d-0215f2408105-kube-api-access-2bchd\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-h79bs\" (UID: \"9b7c74fa-ae1d-441d-923d-0215f2408105\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"
Jul 24 21:20:13.192499 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:13.192474    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"
Jul 24 21:20:13.369541 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:13.369518    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"]
Jul 24 21:20:13.372053 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:20:13.372029    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7c74fa_ae1d_441d_923d_0215f2408105.slice/crio-df573ed87df4fc17e185f68fb8628da5e9c6f5d4e95f0823e8088b1b5484642d WatchSource:0}: Error finding container df573ed87df4fc17e185f68fb8628da5e9c6f5d4e95f0823e8088b1b5484642d: Status 404 returned error can't find the container with id df573ed87df4fc17e185f68fb8628da5e9c6f5d4e95f0823e8088b1b5484642d
Jul 24 21:20:13.973605 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:13.973569    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs" event={"ID":"9b7c74fa-ae1d-441d-923d-0215f2408105","Type":"ContainerStarted","Data":"df573ed87df4fc17e185f68fb8628da5e9c6f5d4e95f0823e8088b1b5484642d"}
Jul 24 21:20:16.989761 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:16.989727    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs" event={"ID":"9b7c74fa-ae1d-441d-923d-0215f2408105","Type":"ContainerStarted","Data":"d7199e99315e528b7b411a35ca10dea37af52f27fdba334c710ebc758cba1139"}
Jul 24 21:20:16.990178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:16.989824    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"
Jul 24 21:20:17.019962 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:17.019907    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs" podStartSLOduration=1.606844874 podStartE2EDuration="5.019883479s" totalImagesPullingTime="3.413038605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:20:17.018832521 +0000 UTC m=+339.050227530" watchObservedRunningTime="2026-07-24 21:20:17.019883479 +0000 UTC m=+339.051278490"
Jul 24 21:20:17.417743 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:17.417710    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:18.436297 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.436267    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"]
Jul 24 21:20:18.454903 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.454877    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"]
Jul 24 21:20:18.455075 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.454956    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:18.457806 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.457766    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 24 21:20:18.458220 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.458204    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 24 21:20:18.458384 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.458365    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zs8xx\""
Jul 24 21:20:18.536259 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.536192    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:18.565407 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.565383    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f67b0c69-309f-4eb9-83b8-ede8284094fe-certificates\") pod \"keda-metrics-apiserver-55557df8c-bghfg\" (UID: \"f67b0c69-309f-4eb9-83b8-ede8284094fe\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:18.565497 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.565433    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795fw\" (UniqueName: \"kubernetes.io/projected/f67b0c69-309f-4eb9-83b8-ede8284094fe-kube-api-access-795fw\") pod \"keda-metrics-apiserver-55557df8c-bghfg\" (UID: \"f67b0c69-309f-4eb9-83b8-ede8284094fe\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:18.565497 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.565473    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f67b0c69-309f-4eb9-83b8-ede8284094fe-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-bghfg\" (UID: \"f67b0c69-309f-4eb9-83b8-ede8284094fe\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:18.662662 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.662638    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:18.662821 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.662727    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-cgw5w"]
Jul 24 21:20:18.664721 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.664706    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-cgw5w"
Jul 24 21:20:18.666994 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.666967    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f67b0c69-309f-4eb9-83b8-ede8284094fe-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-bghfg\" (UID: \"f67b0c69-309f-4eb9-83b8-ede8284094fe\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:18.667113 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.667063    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f67b0c69-309f-4eb9-83b8-ede8284094fe-certificates\") pod \"keda-metrics-apiserver-55557df8c-bghfg\" (UID: \"f67b0c69-309f-4eb9-83b8-ede8284094fe\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:18.667113 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.667089    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795fw\" (UniqueName: \"kubernetes.io/projected/f67b0c69-309f-4eb9-83b8-ede8284094fe-kube-api-access-795fw\") pod \"keda-metrics-apiserver-55557df8c-bghfg\" (UID: \"f67b0c69-309f-4eb9-83b8-ede8284094fe\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:18.667370 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.667352    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f67b0c69-309f-4eb9-83b8-ede8284094fe-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-bghfg\" (UID: \"f67b0c69-309f-4eb9-83b8-ede8284094fe\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:18.667708 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.667693    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 24 21:20:18.669834 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.669815    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f67b0c69-309f-4eb9-83b8-ede8284094fe-certificates\") pod \"keda-metrics-apiserver-55557df8c-bghfg\" (UID: \"f67b0c69-309f-4eb9-83b8-ede8284094fe\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:18.681107 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.681085    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-cgw5w"]
Jul 24 21:20:18.684332 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.684314    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-795fw\" (UniqueName: \"kubernetes.io/projected/f67b0c69-309f-4eb9-83b8-ede8284094fe-kube-api-access-795fw\") pod \"keda-metrics-apiserver-55557df8c-bghfg\" (UID: \"f67b0c69-309f-4eb9-83b8-ede8284094fe\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:18.757204 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.757173    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:18.765103 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.765076    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:18.767826 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.767799    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbdns\" (UniqueName: \"kubernetes.io/projected/8d7eca10-ef63-4771-a806-2dacc864edb5-kube-api-access-zbdns\") pod \"keda-admission-56f9798bb4-cgw5w\" (UID: \"8d7eca10-ef63-4771-a806-2dacc864edb5\") " pod="openshift-keda/keda-admission-56f9798bb4-cgw5w"
Jul 24 21:20:18.767953 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.767923    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d7eca10-ef63-4771-a806-2dacc864edb5-certificates\") pod \"keda-admission-56f9798bb4-cgw5w\" (UID: \"8d7eca10-ef63-4771-a806-2dacc864edb5\") " pod="openshift-keda/keda-admission-56f9798bb4-cgw5w"
Jul 24 21:20:18.859271 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.859196    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:18.869706 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.869642    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d7eca10-ef63-4771-a806-2dacc864edb5-certificates\") pod \"keda-admission-56f9798bb4-cgw5w\" (UID: \"8d7eca10-ef63-4771-a806-2dacc864edb5\") " pod="openshift-keda/keda-admission-56f9798bb4-cgw5w"
Jul 24 21:20:18.869825 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.869765    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbdns\" (UniqueName: \"kubernetes.io/projected/8d7eca10-ef63-4771-a806-2dacc864edb5-kube-api-access-zbdns\") pod \"keda-admission-56f9798bb4-cgw5w\" (UID: \"8d7eca10-ef63-4771-a806-2dacc864edb5\") " pod="openshift-keda/keda-admission-56f9798bb4-cgw5w"
Jul 24 21:20:18.870164 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:20:18.870062    2522 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 24 21:20:18.870164 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:20:18.870105    2522 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-cgw5w: secret "keda-admission-webhooks-certs" not found
Jul 24 21:20:18.870518 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:20:18.870429    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d7eca10-ef63-4771-a806-2dacc864edb5-certificates podName:8d7eca10-ef63-4771-a806-2dacc864edb5 nodeName:}" failed. No retries permitted until 2026-07-24 21:20:19.370391776 +0000 UTC m=+341.401786786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8d7eca10-ef63-4771-a806-2dacc864edb5-certificates") pod "keda-admission-56f9798bb4-cgw5w" (UID: "8d7eca10-ef63-4771-a806-2dacc864edb5") : secret "keda-admission-webhooks-certs" not found
Jul 24 21:20:18.882516 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.882452    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbdns\" (UniqueName: \"kubernetes.io/projected/8d7eca10-ef63-4771-a806-2dacc864edb5-kube-api-access-zbdns\") pod \"keda-admission-56f9798bb4-cgw5w\" (UID: \"8d7eca10-ef63-4771-a806-2dacc864edb5\") " pod="openshift-keda/keda-admission-56f9798bb4-cgw5w"
Jul 24 21:20:18.930981 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:18.930954    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"]
Jul 24 21:20:18.932949 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:20:18.932918    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67b0c69_309f_4eb9_83b8_ede8284094fe.slice/crio-51001aa9a4779394d35b18ecf5d5873819f9f61a8d828c1e819046549eaf9a3d WatchSource:0}: Error finding container 51001aa9a4779394d35b18ecf5d5873819f9f61a8d828c1e819046549eaf9a3d: Status 404 returned error can't find the container with id 51001aa9a4779394d35b18ecf5d5873819f9f61a8d828c1e819046549eaf9a3d
Jul 24 21:20:19.002638 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:19.002603    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg" event={"ID":"f67b0c69-309f-4eb9-83b8-ede8284094fe","Type":"ContainerStarted","Data":"51001aa9a4779394d35b18ecf5d5873819f9f61a8d828c1e819046549eaf9a3d"}
Jul 24 21:20:19.075322 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:19.075238    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:19.180382 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:19.180353    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:19.375124 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:19.375029    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d7eca10-ef63-4771-a806-2dacc864edb5-certificates\") pod \"keda-admission-56f9798bb4-cgw5w\" (UID: \"8d7eca10-ef63-4771-a806-2dacc864edb5\") " pod="openshift-keda/keda-admission-56f9798bb4-cgw5w"
Jul 24 21:20:19.377627 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:19.377604    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d7eca10-ef63-4771-a806-2dacc864edb5-certificates\") pod \"keda-admission-56f9798bb4-cgw5w\" (UID: \"8d7eca10-ef63-4771-a806-2dacc864edb5\") " pod="openshift-keda/keda-admission-56f9798bb4-cgw5w"
Jul 24 21:20:19.483573 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:19.483542    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:19.582509 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:19.582476    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-cgw5w"
Jul 24 21:20:19.607271 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:19.607243    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:19.716314 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:19.716276    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:19.726202 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:19.726112    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-cgw5w"]
Jul 24 21:20:19.728473 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:20:19.728432    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7eca10_ef63_4771_a806_2dacc864edb5.slice/crio-95922decb13deb3cb8efd01be876243eac0ccbf91134ce61a9ad9f6ab1d1e38b WatchSource:0}: Error finding container 95922decb13deb3cb8efd01be876243eac0ccbf91134ce61a9ad9f6ab1d1e38b: Status 404 returned error can't find the container with id 95922decb13deb3cb8efd01be876243eac0ccbf91134ce61a9ad9f6ab1d1e38b
Jul 24 21:20:20.008792 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:20.008752    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-cgw5w" event={"ID":"8d7eca10-ef63-4771-a806-2dacc864edb5","Type":"ContainerStarted","Data":"95922decb13deb3cb8efd01be876243eac0ccbf91134ce61a9ad9f6ab1d1e38b"}
Jul 24 21:20:22.019994 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:22.019956    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-cgw5w" event={"ID":"8d7eca10-ef63-4771-a806-2dacc864edb5","Type":"ContainerStarted","Data":"ee366f3af5915b17e2851e6e60365cb9783af9e87d6b781e61f0974fc53a24a9"}
Jul 24 21:20:22.020455 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:22.020098    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-cgw5w"
Jul 24 21:20:22.021621 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:22.021598    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg" event={"ID":"f67b0c69-309f-4eb9-83b8-ede8284094fe","Type":"ContainerStarted","Data":"9b4532df63a4fdea84337366e6f88940347ca5f08f64c12f068c689972d4c490"}
Jul 24 21:20:22.021751 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:22.021730    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:22.040334 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:22.040285    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-cgw5w" podStartSLOduration=1.8968980229999999 podStartE2EDuration="4.040270116s" totalImagesPullingTime="2.143372093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:20:22.039835573 +0000 UTC m=+344.071230580" watchObservedRunningTime="2026-07-24 21:20:22.040270116 +0000 UTC m=+344.071665126"
Jul 24 21:20:22.059278 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:22.059242    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg" podStartSLOduration=1.118021173 podStartE2EDuration="4.059231815s" totalImagesPullingTime="2.941210642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:20:22.058179761 +0000 UTC m=+344.089574791" watchObservedRunningTime="2026-07-24 21:20:22.059231815 +0000 UTC m=+344.090626836"
Jul 24 21:20:33.029966 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:33.029882    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-bghfg"
Jul 24 21:20:37.995936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:37.995906    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-h79bs"
Jul 24 21:20:43.027823 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:20:43.027785    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-cgw5w"
Jul 24 21:21:43.090575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.090532    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:43.153682 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.153638    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-664687df58-gn457"]
Jul 24 21:21:43.156820 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.156804    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:21:43.160090 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.160068    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-hv9hm\""
Jul 24 21:21:43.160467 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.160448    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 24 21:21:43.160528 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.160503    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 24 21:21:43.160566 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.160503    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 24 21:21:43.166980 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.166226    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-nwfkz"]
Jul 24 21:21:43.172596 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.172567    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-nwfkz"
Jul 24 21:21:43.173290 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.173269    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-gn457"]
Jul 24 21:21:43.174996 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.174969    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 24 21:21:43.175117 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.174982    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-l9dh2\""
Jul 24 21:21:43.176551 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.176527    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-nwfkz"]
Jul 24 21:21:43.180121 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.180097    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:43.279845 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.279822    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6-cert\") pod \"llmisvc-controller-manager-654c557856-nwfkz\" (UID: \"e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6\") " pod="kserve/llmisvc-controller-manager-654c557856-nwfkz"
Jul 24 21:21:43.279952 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.279853    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9t45\" (UniqueName: \"kubernetes.io/projected/e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6-kube-api-access-j9t45\") pod \"llmisvc-controller-manager-654c557856-nwfkz\" (UID: \"e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6\") " pod="kserve/llmisvc-controller-manager-654c557856-nwfkz"
Jul 24 21:21:43.279952 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.279886    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4cc76c0d-4706-46ed-a160-28f767f161fd-cert\") pod \"kserve-controller-manager-664687df58-gn457\" (UID: \"4cc76c0d-4706-46ed-a160-28f767f161fd\") " pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:21:43.279952 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.279943    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc8pk\" (UniqueName: \"kubernetes.io/projected/4cc76c0d-4706-46ed-a160-28f767f161fd-kube-api-access-sc8pk\") pod \"kserve-controller-manager-664687df58-gn457\" (UID: \"4cc76c0d-4706-46ed-a160-28f767f161fd\") " pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:21:43.353393 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.353323    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:43.380923 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.380898    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4cc76c0d-4706-46ed-a160-28f767f161fd-cert\") pod \"kserve-controller-manager-664687df58-gn457\" (UID: \"4cc76c0d-4706-46ed-a160-28f767f161fd\") " pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:21:43.381041 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.380941    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc8pk\" (UniqueName: \"kubernetes.io/projected/4cc76c0d-4706-46ed-a160-28f767f161fd-kube-api-access-sc8pk\") pod \"kserve-controller-manager-664687df58-gn457\" (UID: \"4cc76c0d-4706-46ed-a160-28f767f161fd\") " pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:21:43.381041 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.380985    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6-cert\") pod \"llmisvc-controller-manager-654c557856-nwfkz\" (UID: \"e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6\") " pod="kserve/llmisvc-controller-manager-654c557856-nwfkz"
Jul 24 21:21:43.381041 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.381013    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9t45\" (UniqueName: \"kubernetes.io/projected/e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6-kube-api-access-j9t45\") pod \"llmisvc-controller-manager-654c557856-nwfkz\" (UID: \"e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6\") " pod="kserve/llmisvc-controller-manager-654c557856-nwfkz"
Jul 24 21:21:43.383473 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.383445    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4cc76c0d-4706-46ed-a160-28f767f161fd-cert\") pod \"kserve-controller-manager-664687df58-gn457\" (UID: \"4cc76c0d-4706-46ed-a160-28f767f161fd\") " pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:21:43.383557 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.383450    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6-cert\") pod \"llmisvc-controller-manager-654c557856-nwfkz\" (UID: \"e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6\") " pod="kserve/llmisvc-controller-manager-654c557856-nwfkz"
Jul 24 21:21:43.396551 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.396524    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc8pk\" (UniqueName: \"kubernetes.io/projected/4cc76c0d-4706-46ed-a160-28f767f161fd-kube-api-access-sc8pk\") pod \"kserve-controller-manager-664687df58-gn457\" (UID: \"4cc76c0d-4706-46ed-a160-28f767f161fd\") " pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:21:43.396711 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.396693    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9t45\" (UniqueName: \"kubernetes.io/projected/e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6-kube-api-access-j9t45\") pod \"llmisvc-controller-manager-654c557856-nwfkz\" (UID: \"e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6\") " pod="kserve/llmisvc-controller-manager-654c557856-nwfkz"
Jul 24 21:21:43.461451 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.461424    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:43.472922 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.472898    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:21:43.482888 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.482870    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-nwfkz"
Jul 24 21:21:43.644920 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.644898    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-gn457"]
Jul 24 21:21:43.646753 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:21:43.646730    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc76c0d_4706_46ed_a160_28f767f161fd.slice/crio-c28e5cc499c6ba385050571cd654d6723a07071d31ab2600986cc0ca85bbc128 WatchSource:0}: Error finding container c28e5cc499c6ba385050571cd654d6723a07071d31ab2600986cc0ca85bbc128: Status 404 returned error can't find the container with id c28e5cc499c6ba385050571cd654d6723a07071d31ab2600986cc0ca85bbc128
Jul 24 21:21:43.671633 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.671612    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-nwfkz"]
Jul 24 21:21:43.674286 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:21:43.674251    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode31f3f1c_99c8_4d58_bc93_2b6b8521e2d6.slice/crio-9ca2fa46aced481efe72dcfcc4176c31d2be161db4fff94dab432e8371f9e743 WatchSource:0}: Error finding container 9ca2fa46aced481efe72dcfcc4176c31d2be161db4fff94dab432e8371f9e743: Status 404 returned error can't find the container with id 9ca2fa46aced481efe72dcfcc4176c31d2be161db4fff94dab432e8371f9e743
Jul 24 21:21:43.862176 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.862147    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:43.964716 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:43.964667    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:44.064677 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:44.064656    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:44.389577 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:44.389467    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-nwfkz" event={"ID":"e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6","Type":"ContainerStarted","Data":"9ca2fa46aced481efe72dcfcc4176c31d2be161db4fff94dab432e8371f9e743"}
Jul 24 21:21:44.391191 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:44.391149    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-gn457" event={"ID":"4cc76c0d-4706-46ed-a160-28f767f161fd","Type":"ContainerStarted","Data":"c28e5cc499c6ba385050571cd654d6723a07071d31ab2600986cc0ca85bbc128"}
Jul 24 21:21:47.406999 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:47.406959    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-nwfkz" event={"ID":"e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6","Type":"ContainerStarted","Data":"83c64655daa13d0637cf02a7ceed3aa797f2017825ba0ae7636d5d56c3d7849a"}
Jul 24 21:21:47.407430 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:47.407081    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-nwfkz"
Jul 24 21:21:47.408464 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:47.408439    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-gn457" event={"ID":"4cc76c0d-4706-46ed-a160-28f767f161fd","Type":"ContainerStarted","Data":"27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f"}
Jul 24 21:21:47.408562 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:47.408500    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:21:47.434208 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:47.434166    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-nwfkz" podStartSLOduration=1.187237978 podStartE2EDuration="4.434154458s" totalImagesPullingTime="3.24691648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:21:47.433631268 +0000 UTC m=+429.465026276" watchObservedRunningTime="2026-07-24 21:21:47.434154458 +0000 UTC m=+429.465549466"
Jul 24 21:21:47.463178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:21:47.463137    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-664687df58-gn457" podStartSLOduration=1.225526994 podStartE2EDuration="4.463124903s" totalImagesPullingTime="3.237597909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:21:47.461989801 +0000 UTC m=+429.493384808" watchObservedRunningTime="2026-07-24 21:21:47.463124903 +0000 UTC m=+429.494519976"
Jul 24 21:22:03.815584 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:03.815472    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:03.923828 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:03.923794    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:14.942219 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:14.942186    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-548f886447-zxwfc"]
Jul 24 21:22:15.105794 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:15.105745    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:15.207335 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:15.207266    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:18.414346 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:18.414307    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-nwfkz"
Jul 24 21:22:18.417425 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:18.417399    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:22:30.574806 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.574770    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-664687df58-gn457"]
Jul 24 21:22:30.575322 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.574971    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-664687df58-gn457" podUID="4cc76c0d-4706-46ed-a160-28f767f161fd" containerName="manager" containerID="cri-o://27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f" gracePeriod=10
Jul 24 21:22:30.678495 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.678466    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-664687df58-584jx"]
Jul 24 21:22:30.680775 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.680759    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-584jx"
Jul 24 21:22:30.710349 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.710315    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f0a1a16-9f43-4cad-96f6-9ba84516abdf-cert\") pod \"kserve-controller-manager-664687df58-584jx\" (UID: \"2f0a1a16-9f43-4cad-96f6-9ba84516abdf\") " pod="kserve/kserve-controller-manager-664687df58-584jx"
Jul 24 21:22:30.710501 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.710366    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zt2b\" (UniqueName: \"kubernetes.io/projected/2f0a1a16-9f43-4cad-96f6-9ba84516abdf-kube-api-access-5zt2b\") pod \"kserve-controller-manager-664687df58-584jx\" (UID: \"2f0a1a16-9f43-4cad-96f6-9ba84516abdf\") " pod="kserve/kserve-controller-manager-664687df58-584jx"
Jul 24 21:22:30.713716 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.713658    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-584jx"]
Jul 24 21:22:30.811057 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.811034    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zt2b\" (UniqueName: \"kubernetes.io/projected/2f0a1a16-9f43-4cad-96f6-9ba84516abdf-kube-api-access-5zt2b\") pod \"kserve-controller-manager-664687df58-584jx\" (UID: \"2f0a1a16-9f43-4cad-96f6-9ba84516abdf\") " pod="kserve/kserve-controller-manager-664687df58-584jx"
Jul 24 21:22:30.811181 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.811136    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f0a1a16-9f43-4cad-96f6-9ba84516abdf-cert\") pod \"kserve-controller-manager-664687df58-584jx\" (UID: \"2f0a1a16-9f43-4cad-96f6-9ba84516abdf\") " pod="kserve/kserve-controller-manager-664687df58-584jx"
Jul 24 21:22:30.813466 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.813443    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f0a1a16-9f43-4cad-96f6-9ba84516abdf-cert\") pod \"kserve-controller-manager-664687df58-584jx\" (UID: \"2f0a1a16-9f43-4cad-96f6-9ba84516abdf\") " pod="kserve/kserve-controller-manager-664687df58-584jx"
Jul 24 21:22:30.816320 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.816302    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:22:30.835439 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.835388    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zt2b\" (UniqueName: \"kubernetes.io/projected/2f0a1a16-9f43-4cad-96f6-9ba84516abdf-kube-api-access-5zt2b\") pod \"kserve-controller-manager-664687df58-584jx\" (UID: \"2f0a1a16-9f43-4cad-96f6-9ba84516abdf\") " pod="kserve/kserve-controller-manager-664687df58-584jx"
Jul 24 21:22:30.912108 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.912084    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4cc76c0d-4706-46ed-a160-28f767f161fd-cert\") pod \"4cc76c0d-4706-46ed-a160-28f767f161fd\" (UID: \"4cc76c0d-4706-46ed-a160-28f767f161fd\") "
Jul 24 21:22:30.912233 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.912172    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc8pk\" (UniqueName: \"kubernetes.io/projected/4cc76c0d-4706-46ed-a160-28f767f161fd-kube-api-access-sc8pk\") pod \"4cc76c0d-4706-46ed-a160-28f767f161fd\" (UID: \"4cc76c0d-4706-46ed-a160-28f767f161fd\") "
Jul 24 21:22:30.914188 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.914163    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc76c0d-4706-46ed-a160-28f767f161fd-cert" (OuterVolumeSpecName: "cert") pod "4cc76c0d-4706-46ed-a160-28f767f161fd" (UID: "4cc76c0d-4706-46ed-a160-28f767f161fd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:22:30.914290 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.914207    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc76c0d-4706-46ed-a160-28f767f161fd-kube-api-access-sc8pk" (OuterVolumeSpecName: "kube-api-access-sc8pk") pod "4cc76c0d-4706-46ed-a160-28f767f161fd" (UID: "4cc76c0d-4706-46ed-a160-28f767f161fd"). InnerVolumeSpecName "kube-api-access-sc8pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:22:30.991193 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:30.991157    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-584jx"
Jul 24 21:22:31.013489 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.013461    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc8pk\" (UniqueName: \"kubernetes.io/projected/4cc76c0d-4706-46ed-a160-28f767f161fd-kube-api-access-sc8pk\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:22:31.013587 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.013494    2522 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4cc76c0d-4706-46ed-a160-28f767f161fd-cert\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:22:31.130190 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.130117    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-584jx"]
Jul 24 21:22:31.133442 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:22:31.133416    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0a1a16_9f43_4cad_96f6_9ba84516abdf.slice/crio-3cee3e6f7302c29607db182bf281daf589821208faefc53801e2dbde5863fecf WatchSource:0}: Error finding container 3cee3e6f7302c29607db182bf281daf589821208faefc53801e2dbde5863fecf: Status 404 returned error can't find the container with id 3cee3e6f7302c29607db182bf281daf589821208faefc53801e2dbde5863fecf
Jul 24 21:22:31.604341 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.604309    2522 generic.go:356] "Generic (PLEG): container finished" podID="4cc76c0d-4706-46ed-a160-28f767f161fd" containerID="27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f" exitCode=0
Jul 24 21:22:31.604737 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.604381    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-gn457"
Jul 24 21:22:31.604737 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.604384    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-gn457" event={"ID":"4cc76c0d-4706-46ed-a160-28f767f161fd","Type":"ContainerDied","Data":"27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f"}
Jul 24 21:22:31.604737 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.604498    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-gn457" event={"ID":"4cc76c0d-4706-46ed-a160-28f767f161fd","Type":"ContainerDied","Data":"c28e5cc499c6ba385050571cd654d6723a07071d31ab2600986cc0ca85bbc128"}
Jul 24 21:22:31.604737 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.604524    2522 scope.go:117] "RemoveContainer" containerID="27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f"
Jul 24 21:22:31.606099 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.606057    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-584jx" event={"ID":"2f0a1a16-9f43-4cad-96f6-9ba84516abdf","Type":"ContainerStarted","Data":"918f3efb3c3cb7edc034ae1c23dc21cf491caa6b6f76abef06c5074c563f30e5"}
Jul 24 21:22:31.606296 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.606107    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-584jx" event={"ID":"2f0a1a16-9f43-4cad-96f6-9ba84516abdf","Type":"ContainerStarted","Data":"3cee3e6f7302c29607db182bf281daf589821208faefc53801e2dbde5863fecf"}
Jul 24 21:22:31.606296 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.606134    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-664687df58-584jx"
Jul 24 21:22:31.614180 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.614164    2522 scope.go:117] "RemoveContainer" containerID="27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f"
Jul 24 21:22:31.614446 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:22:31.614429    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f\": container with ID starting with 27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f not found: ID does not exist" containerID="27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f"
Jul 24 21:22:31.614516 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.614453    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f"} err="failed to get container status \"27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f\": rpc error: code = NotFound desc = could not find container \"27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f\": container with ID starting with 27957ca9ad3d0a04163bb9d73c9778275e419279d8344f8c7c4557fecbd3059f not found: ID does not exist"
Jul 24 21:22:31.644770 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.644729    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-664687df58-584jx" podStartSLOduration=1.286921489 podStartE2EDuration="1.64471729s" totalImagesPullingTime="357.795801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:22:31.643349876 +0000 UTC m=+473.674744884" watchObservedRunningTime="2026-07-24 21:22:31.64471729 +0000 UTC m=+473.676112298"
Jul 24 21:22:31.670927 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.670902    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-664687df58-gn457"]
Jul 24 21:22:31.682768 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:31.682750    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-664687df58-gn457"]
Jul 24 21:22:32.407508 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:32.407476    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc76c0d-4706-46ed-a160-28f767f161fd" path="/var/lib/kubelet/pods/4cc76c0d-4706-46ed-a160-28f767f161fd/volumes"
Jul 24 21:22:39.961928 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:39.961880    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-548f886447-zxwfc" podUID="ea7c8b57-df75-4cbc-a4f2-6734db324d84" containerName="console" containerID="cri-o://8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6" gracePeriod=15
Jul 24 21:22:40.213559 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.213507    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:22:40.286375 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.286349    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-trusted-ca-bundle\") pod \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") "
Jul 24 21:22:40.286504 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.286437    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-config\") pod \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") "
Jul 24 21:22:40.286504 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.286468    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-oauth-serving-cert\") pod \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") "
Jul 24 21:22:40.286578 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.286503    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szxpq\" (UniqueName: \"kubernetes.io/projected/ea7c8b57-df75-4cbc-a4f2-6734db324d84-kube-api-access-szxpq\") pod \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") "
Jul 24 21:22:40.286578 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.286537    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-serving-cert\") pod \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") "
Jul 24 21:22:40.286654 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.286590    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-oauth-config\") pod \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") "
Jul 24 21:22:40.286654 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.286617    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-service-ca\") pod \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\" (UID: \"ea7c8b57-df75-4cbc-a4f2-6734db324d84\") "
Jul 24 21:22:40.286814 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.286774    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ea7c8b57-df75-4cbc-a4f2-6734db324d84" (UID: "ea7c8b57-df75-4cbc-a4f2-6734db324d84"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:22:40.286919 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.286894    2522 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-trusted-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:22:40.286919 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.286900    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-config" (OuterVolumeSpecName: "console-config") pod "ea7c8b57-df75-4cbc-a4f2-6734db324d84" (UID: "ea7c8b57-df75-4cbc-a4f2-6734db324d84"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:22:40.286919 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.286908    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ea7c8b57-df75-4cbc-a4f2-6734db324d84" (UID: "ea7c8b57-df75-4cbc-a4f2-6734db324d84"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:22:40.287148 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.287126    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-service-ca" (OuterVolumeSpecName: "service-ca") pod "ea7c8b57-df75-4cbc-a4f2-6734db324d84" (UID: "ea7c8b57-df75-4cbc-a4f2-6734db324d84"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:22:40.288769 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.288742    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ea7c8b57-df75-4cbc-a4f2-6734db324d84" (UID: "ea7c8b57-df75-4cbc-a4f2-6734db324d84"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:22:40.288856 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.288829    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7c8b57-df75-4cbc-a4f2-6734db324d84-kube-api-access-szxpq" (OuterVolumeSpecName: "kube-api-access-szxpq") pod "ea7c8b57-df75-4cbc-a4f2-6734db324d84" (UID: "ea7c8b57-df75-4cbc-a4f2-6734db324d84"). InnerVolumeSpecName "kube-api-access-szxpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:22:40.289084 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.289069    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ea7c8b57-df75-4cbc-a4f2-6734db324d84" (UID: "ea7c8b57-df75-4cbc-a4f2-6734db324d84"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:22:40.387180 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.387149    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szxpq\" (UniqueName: \"kubernetes.io/projected/ea7c8b57-df75-4cbc-a4f2-6734db324d84-kube-api-access-szxpq\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:22:40.387180 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.387179    2522 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-serving-cert\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:22:40.387180 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.387188    2522 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-oauth-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:22:40.387337 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.387197    2522 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-service-ca\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:22:40.387337 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.387206    2522 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-console-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:22:40.387337 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.387213    2522 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea7c8b57-df75-4cbc-a4f2-6734db324d84-oauth-serving-cert\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:22:40.652385 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.652300    2522 generic.go:356] "Generic (PLEG): container finished" podID="ea7c8b57-df75-4cbc-a4f2-6734db324d84" containerID="8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6" exitCode=0
Jul 24 21:22:40.652385 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.652371    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-548f886447-zxwfc"
Jul 24 21:22:40.652595 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.652391    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-548f886447-zxwfc" event={"ID":"ea7c8b57-df75-4cbc-a4f2-6734db324d84","Type":"ContainerDied","Data":"8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6"}
Jul 24 21:22:40.652595 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.652471    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-548f886447-zxwfc" event={"ID":"ea7c8b57-df75-4cbc-a4f2-6734db324d84","Type":"ContainerDied","Data":"9c4c8abcbe7791cb446790a86f18cb3386f2cf10f989834413ec02158d354dcc"}
Jul 24 21:22:40.652595 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.652492    2522 scope.go:117] "RemoveContainer" containerID="8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6"
Jul 24 21:22:40.661414 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.661397    2522 scope.go:117] "RemoveContainer" containerID="8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6"
Jul 24 21:22:40.661788 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:22:40.661770    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6\": container with ID starting with 8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6 not found: ID does not exist" containerID="8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6"
Jul 24 21:22:40.661858 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.661800    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6"} err="failed to get container status \"8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6\": rpc error: code = NotFound desc = could not find container \"8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6\": container with ID starting with 8d6b5d3374298983055bff77b8f47e9a33ec72e5a040c8c58bc37753a38f25e6 not found: ID does not exist"
Jul 24 21:22:40.669112 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.669090    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-548f886447-zxwfc"]
Jul 24 21:22:40.672762 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:40.672742    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-548f886447-zxwfc"]
Jul 24 21:22:42.407296 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:22:42.407256    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea7c8b57-df75-4cbc-a4f2-6734db324d84" path="/var/lib/kubelet/pods/ea7c8b57-df75-4cbc-a4f2-6734db324d84/volumes"
Jul 24 21:23:02.618403 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:23:02.618372    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-664687df58-584jx"
Jul 24 21:23:03.858785 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:23:03.858739    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:04.023260 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:23:04.023227    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:04.136495 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:23:04.136407    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:05.040685 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:23:05.040647    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:05.146360 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:23:05.146322    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:15.045827 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:23:15.045792    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:19.291138 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:23:19.291100    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:15.599269 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:15.599228    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:15.657664 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:15.657623    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:39.867118 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:39.867085    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:39.967220 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:39.967190    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:51.455829 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.455794    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:51.680029 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.679996    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:51.780301 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.780220    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:51.897410 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.897371    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:51.906816 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.906789    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"]
Jul 24 21:24:51.907123 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.907111    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7c8b57-df75-4cbc-a4f2-6734db324d84" containerName="console"
Jul 24 21:24:51.907168 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.907125    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7c8b57-df75-4cbc-a4f2-6734db324d84" containerName="console"
Jul 24 21:24:51.907168 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.907148    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc76c0d-4706-46ed-a160-28f767f161fd" containerName="manager"
Jul 24 21:24:51.907168 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.907153    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc76c0d-4706-46ed-a160-28f767f161fd" containerName="manager"
Jul 24 21:24:51.907257 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.907212    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc76c0d-4706-46ed-a160-28f767f161fd" containerName="manager"
Jul 24 21:24:51.907257 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.907221    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea7c8b57-df75-4cbc-a4f2-6734db324d84" containerName="console"
Jul 24 21:24:51.910147 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.910132    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:24:51.913038 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.913020    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-78szr\""
Jul 24 21:24:51.913268 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.913253    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-77ecc-kube-rbac-proxy-sar-config\""
Jul 24 21:24:51.913347 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.913251    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-77ecc-predictor-serving-cert\""
Jul 24 21:24:51.913836 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.913820    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:24:51.913904 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.913849    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:24:51.923869 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:51.923850    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"]
Jul 24 21:24:52.007074 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.007047    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:52.043292 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.043221    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ff6f332-3d37-402a-a5b4-d35b75b85475-proxy-tls\") pod \"error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") " pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:24:52.043411 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.043291    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") " pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:24:52.043411 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.043336    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tbdg\" (UniqueName: \"kubernetes.io/projected/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-api-access-7tbdg\") pod \"error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") " pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:24:52.119017 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.118982    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:52.143974 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.143950    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") " pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:24:52.144095 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.143986    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tbdg\" (UniqueName: \"kubernetes.io/projected/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-api-access-7tbdg\") pod \"error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") " pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:24:52.144095 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.144025    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ff6f332-3d37-402a-a5b4-d35b75b85475-proxy-tls\") pod \"error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") " pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:24:52.144554 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.144536    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") " pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:24:52.146480 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.146462    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ff6f332-3d37-402a-a5b4-d35b75b85475-proxy-tls\") pod \"error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") " pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:24:52.154399 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.154375    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tbdg\" (UniqueName: \"kubernetes.io/projected/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-api-access-7tbdg\") pod \"error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") " pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:24:52.220896 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.220872    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:24:52.237738 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.237706    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:52.349225 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.349196    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:52.359285 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.359264    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"]
Jul 24 21:24:52.362159 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:24:52.362132    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff6f332_3d37_402a_a5b4_d35b75b85475.slice/crio-623ffdfa0b34b1cb822fab942b972f9624970f1318ce7df37b3028fbdf5a3f3a WatchSource:0}: Error finding container 623ffdfa0b34b1cb822fab942b972f9624970f1318ce7df37b3028fbdf5a3f3a: Status 404 returned error can't find the container with id 623ffdfa0b34b1cb822fab942b972f9624970f1318ce7df37b3028fbdf5a3f3a
Jul 24 21:24:52.364245 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.364227    2522 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:24:52.449789 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.449751    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:52.601725 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:52.601636    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:53.256223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:24:53.256144    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" event={"ID":"6ff6f332-3d37-402a-a5b4-d35b75b85475","Type":"ContainerStarted","Data":"623ffdfa0b34b1cb822fab942b972f9624970f1318ce7df37b3028fbdf5a3f3a"}
Jul 24 21:25:04.320418 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:04.320381    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" event={"ID":"6ff6f332-3d37-402a-a5b4-d35b75b85475","Type":"ContainerStarted","Data":"c7804d12fed6ce9c1b0392c50945646e7d5e1916b958545e6aa94ff74ff0f1a2"}
Jul 24 21:25:06.331521 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:06.331434    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" event={"ID":"6ff6f332-3d37-402a-a5b4-d35b75b85475","Type":"ContainerStarted","Data":"b9461226a834b50d21911680e3a140c7e3e93c0bde059f35f31ccdd5f3eac278"}
Jul 24 21:25:06.331908 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:06.331561    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:25:06.350255 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:06.350206    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" podStartSLOduration=1.704882523 podStartE2EDuration="15.350188408s" totalImagesPullingTime="13.645305885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:24:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:25:06.348769729 +0000 UTC m=+628.380164739" watchObservedRunningTime="2026-07-24 21:25:06.350188408 +0000 UTC m=+628.381583416"
Jul 24 21:25:07.335702 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:07.335657    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:25:07.336664 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:07.336632    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 24 21:25:08.339826 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:08.339787    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 24 21:25:13.344218 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:13.344187    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:25:13.344706 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:13.344633    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 24 21:25:23.345103 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:23.345062    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 24 21:25:33.345474 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:33.345438    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 24 21:25:43.345323 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:43.345284    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 24 21:25:53.345469 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:25:53.345430    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:26:11.504124 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.504091    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"]
Jul 24 21:26:11.507482 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.507464    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:11.510345 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.510324    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-77ecc-kube-rbac-proxy-sar-config\""
Jul 24 21:26:11.510466 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.510330    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-77ecc-serving-cert\""
Jul 24 21:26:11.515416 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.515375    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:11.516059 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.516044    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"]
Jul 24 21:26:11.641061 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.641023    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d76e551-a8c7-43fa-81c0-a176fee9474a-openshift-service-ca-bundle\") pod \"switch-graph-77ecc-597566fc6b-4bnvg\" (UID: \"9d76e551-a8c7-43fa-81c0-a176fee9474a\") " pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:11.641218 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.641078    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d76e551-a8c7-43fa-81c0-a176fee9474a-proxy-tls\") pod \"switch-graph-77ecc-597566fc6b-4bnvg\" (UID: \"9d76e551-a8c7-43fa-81c0-a176fee9474a\") " pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:11.742420 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.742390    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d76e551-a8c7-43fa-81c0-a176fee9474a-proxy-tls\") pod \"switch-graph-77ecc-597566fc6b-4bnvg\" (UID: \"9d76e551-a8c7-43fa-81c0-a176fee9474a\") " pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:11.742538 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.742454    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d76e551-a8c7-43fa-81c0-a176fee9474a-openshift-service-ca-bundle\") pod \"switch-graph-77ecc-597566fc6b-4bnvg\" (UID: \"9d76e551-a8c7-43fa-81c0-a176fee9474a\") " pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:11.743089 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.743071    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d76e551-a8c7-43fa-81c0-a176fee9474a-openshift-service-ca-bundle\") pod \"switch-graph-77ecc-597566fc6b-4bnvg\" (UID: \"9d76e551-a8c7-43fa-81c0-a176fee9474a\") " pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:11.744973 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.744955    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d76e551-a8c7-43fa-81c0-a176fee9474a-proxy-tls\") pod \"switch-graph-77ecc-597566fc6b-4bnvg\" (UID: \"9d76e551-a8c7-43fa-81c0-a176fee9474a\") " pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:11.819032 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.818981    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:11.948405 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:11.948374    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"]
Jul 24 21:26:11.951218 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:26:11.951184    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d76e551_a8c7_43fa_81c0_a176fee9474a.slice/crio-886dfb9ad8522d783b0320dc846e73c9031e98d81e4897f2d9d8dab8c0868472 WatchSource:0}: Error finding container 886dfb9ad8522d783b0320dc846e73c9031e98d81e4897f2d9d8dab8c0868472: Status 404 returned error can't find the container with id 886dfb9ad8522d783b0320dc846e73c9031e98d81e4897f2d9d8dab8c0868472
Jul 24 21:26:12.618630 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:12.618593    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" event={"ID":"9d76e551-a8c7-43fa-81c0-a176fee9474a","Type":"ContainerStarted","Data":"886dfb9ad8522d783b0320dc846e73c9031e98d81e4897f2d9d8dab8c0868472"}
Jul 24 21:26:12.747331 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:12.747295    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:14.630205 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:14.630167    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" event={"ID":"9d76e551-a8c7-43fa-81c0-a176fee9474a","Type":"ContainerStarted","Data":"4ffdef2510b12910f3bc5ea331680df104f8d94419ab4c0ca1ff2429975017a4"}
Jul 24 21:26:14.630618 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:14.630308    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:14.653215 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:14.649212    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" podStartSLOduration=1.5626394399999999 podStartE2EDuration="3.649192076s" totalImagesPullingTime="2.086552636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:26:14.645926562 +0000 UTC m=+696.677321582" watchObservedRunningTime="2026-07-24 21:26:14.649192076 +0000 UTC m=+696.680587086"
Jul 24 21:26:20.642493 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:20.642460    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:21.683510 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:21.683480    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"]
Jul 24 21:26:21.683921 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:21.683654    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" podUID="9d76e551-a8c7-43fa-81c0-a176fee9474a" containerName="switch-graph-77ecc" containerID="cri-o://4ffdef2510b12910f3bc5ea331680df104f8d94419ab4c0ca1ff2429975017a4" gracePeriod=30
Jul 24 21:26:21.706622 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:21.706575    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:21.843212 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:21.843165    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:21.876594 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:21.876566    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"]
Jul 24 21:26:21.876858 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:21.876828    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kserve-container" containerID="cri-o://c7804d12fed6ce9c1b0392c50945646e7d5e1916b958545e6aa94ff74ff0f1a2" gracePeriod=30
Jul 24 21:26:21.876952 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:21.876912    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kube-rbac-proxy" containerID="cri-o://b9461226a834b50d21911680e3a140c7e3e93c0bde059f35f31ccdd5f3eac278" gracePeriod=30
Jul 24 21:26:21.938739 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:21.938715    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:21.995006 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:21.994987    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"]
Jul 24 21:26:21.998250 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:21.998231    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.000518 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.000495    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eea53-kube-rbac-proxy-sar-config\""
Jul 24 21:26:22.000615 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.000526    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eea53-predictor-serving-cert\""
Jul 24 21:26:22.008793 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.008772    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"]
Jul 24 21:26:22.024177 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.024149    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:22.116044 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.116017    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c798a1f6-74e8-48e5-ae62-fc1a546d596a-proxy-tls\") pod \"error-404-isvc-eea53-predictor-6bd495b56f-7z42x\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") " pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.116182 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.116059    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rhx\" (UniqueName: \"kubernetes.io/projected/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-api-access-59rhx\") pod \"error-404-isvc-eea53-predictor-6bd495b56f-7z42x\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") " pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.116182 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.116114    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eea53-predictor-6bd495b56f-7z42x\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") " pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.134001 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.133977    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:22.217184 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.217114    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c798a1f6-74e8-48e5-ae62-fc1a546d596a-proxy-tls\") pod \"error-404-isvc-eea53-predictor-6bd495b56f-7z42x\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") " pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.217184 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.217146    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rhx\" (UniqueName: \"kubernetes.io/projected/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-api-access-59rhx\") pod \"error-404-isvc-eea53-predictor-6bd495b56f-7z42x\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") " pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.217184 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.217167    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eea53-predictor-6bd495b56f-7z42x\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") " pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.217898 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.217880    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eea53-predictor-6bd495b56f-7z42x\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") " pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.219766 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.219747    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c798a1f6-74e8-48e5-ae62-fc1a546d596a-proxy-tls\") pod \"error-404-isvc-eea53-predictor-6bd495b56f-7z42x\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") " pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.233344 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.233320    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rhx\" (UniqueName: \"kubernetes.io/projected/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-api-access-59rhx\") pod \"error-404-isvc-eea53-predictor-6bd495b56f-7z42x\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") " pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.309017 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.308995    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.327487 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.327465    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:22.444239 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.444216    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"]
Jul 24 21:26:22.445956 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:26:22.445923    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc798a1f6_74e8_48e5_ae62_fc1a546d596a.slice/crio-f8ec224bfa4812fdc0f6e12eb62441daca561cb4ba220784bd4fa9a95f7c53cf WatchSource:0}: Error finding container f8ec224bfa4812fdc0f6e12eb62441daca561cb4ba220784bd4fa9a95f7c53cf: Status 404 returned error can't find the container with id f8ec224bfa4812fdc0f6e12eb62441daca561cb4ba220784bd4fa9a95f7c53cf
Jul 24 21:26:22.672885 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.672835    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" event={"ID":"c798a1f6-74e8-48e5-ae62-fc1a546d596a","Type":"ContainerStarted","Data":"af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944"}
Jul 24 21:26:22.673055 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.672982    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.673055 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.673044    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" event={"ID":"c798a1f6-74e8-48e5-ae62-fc1a546d596a","Type":"ContainerStarted","Data":"9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8"}
Jul 24 21:26:22.673165 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.673083    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" event={"ID":"c798a1f6-74e8-48e5-ae62-fc1a546d596a","Type":"ContainerStarted","Data":"f8ec224bfa4812fdc0f6e12eb62441daca561cb4ba220784bd4fa9a95f7c53cf"}
Jul 24 21:26:22.673165 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.673117    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:22.674716 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.674688    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 24 21:26:22.674923 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.674903    2522 generic.go:356] "Generic (PLEG): container finished" podID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerID="b9461226a834b50d21911680e3a140c7e3e93c0bde059f35f31ccdd5f3eac278" exitCode=2
Jul 24 21:26:22.674973 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.674959    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" event={"ID":"6ff6f332-3d37-402a-a5b4-d35b75b85475","Type":"ContainerDied","Data":"b9461226a834b50d21911680e3a140c7e3e93c0bde059f35f31ccdd5f3eac278"}
Jul 24 21:26:22.693321 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:22.693075    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" podStartSLOduration=1.693056992 podStartE2EDuration="1.693056992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:26:22.691969877 +0000 UTC m=+704.723364886" watchObservedRunningTime="2026-07-24 21:26:22.693056992 +0000 UTC m=+704.724452001"
Jul 24 21:26:23.340943 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:23.340907    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 24 21:26:23.345299 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:23.345271    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 24 21:26:23.679653 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:23.679551    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 24 21:26:24.685968 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.685932    2522 generic.go:356] "Generic (PLEG): container finished" podID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerID="c7804d12fed6ce9c1b0392c50945646e7d5e1916b958545e6aa94ff74ff0f1a2" exitCode=0
Jul 24 21:26:24.686333 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.685964    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" event={"ID":"6ff6f332-3d37-402a-a5b4-d35b75b85475","Type":"ContainerDied","Data":"c7804d12fed6ce9c1b0392c50945646e7d5e1916b958545e6aa94ff74ff0f1a2"}
Jul 24 21:26:24.728542 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.728522    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:26:24.838317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.838246    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-rbac-proxy-sar-config\") pod \"6ff6f332-3d37-402a-a5b4-d35b75b85475\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") "
Jul 24 21:26:24.838317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.838275    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tbdg\" (UniqueName: \"kubernetes.io/projected/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-api-access-7tbdg\") pod \"6ff6f332-3d37-402a-a5b4-d35b75b85475\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") "
Jul 24 21:26:24.838317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.838303    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ff6f332-3d37-402a-a5b4-d35b75b85475-proxy-tls\") pod \"6ff6f332-3d37-402a-a5b4-d35b75b85475\" (UID: \"6ff6f332-3d37-402a-a5b4-d35b75b85475\") "
Jul 24 21:26:24.838661 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.838639    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ff6f332-3d37-402a-a5b4-d35b75b85475" (UID: "6ff6f332-3d37-402a-a5b4-d35b75b85475"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:26:24.840536 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.840509    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff6f332-3d37-402a-a5b4-d35b75b85475-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ff6f332-3d37-402a-a5b4-d35b75b85475" (UID: "6ff6f332-3d37-402a-a5b4-d35b75b85475"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:26:24.840632 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.840590    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-api-access-7tbdg" (OuterVolumeSpecName: "kube-api-access-7tbdg") pod "6ff6f332-3d37-402a-a5b4-d35b75b85475" (UID: "6ff6f332-3d37-402a-a5b4-d35b75b85475"). InnerVolumeSpecName "kube-api-access-7tbdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:26:24.939798 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.939765    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:26:24.939798 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.939792    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tbdg\" (UniqueName: \"kubernetes.io/projected/6ff6f332-3d37-402a-a5b4-d35b75b85475-kube-api-access-7tbdg\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:26:24.939798 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:24.939802    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ff6f332-3d37-402a-a5b4-d35b75b85475-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:26:25.639511 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:25.639478    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" podUID="9d76e551-a8c7-43fa-81c0-a176fee9474a" containerName="switch-graph-77ecc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:25.692973 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:25.692938    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"
Jul 24 21:26:25.693367 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:25.692941    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq" event={"ID":"6ff6f332-3d37-402a-a5b4-d35b75b85475","Type":"ContainerDied","Data":"623ffdfa0b34b1cb822fab942b972f9624970f1318ce7df37b3028fbdf5a3f3a"}
Jul 24 21:26:25.693367 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:25.693089    2522 scope.go:117] "RemoveContainer" containerID="b9461226a834b50d21911680e3a140c7e3e93c0bde059f35f31ccdd5f3eac278"
Jul 24 21:26:25.702262 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:25.702242    2522 scope.go:117] "RemoveContainer" containerID="c7804d12fed6ce9c1b0392c50945646e7d5e1916b958545e6aa94ff74ff0f1a2"
Jul 24 21:26:25.716739 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:25.716702    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"]
Jul 24 21:26:25.720743 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:25.720716    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77ecc-predictor-7bcd8f7bd9-vdcnq"]
Jul 24 21:26:26.407758 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:26.407730    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" path="/var/lib/kubelet/pods/6ff6f332-3d37-402a-a5b4-d35b75b85475/volumes"
Jul 24 21:26:28.684156 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:28.684127    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:26:28.684731 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:28.684708    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 24 21:26:30.638986 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:30.638903    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" podUID="9d76e551-a8c7-43fa-81c0-a176fee9474a" containerName="switch-graph-77ecc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:35.639292 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:35.639258    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" podUID="9d76e551-a8c7-43fa-81c0-a176fee9474a" containerName="switch-graph-77ecc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:35.639864 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:35.639368    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:38.685588 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:38.685502    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 24 21:26:40.638848 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:40.638805    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" podUID="9d76e551-a8c7-43fa-81c0-a176fee9474a" containerName="switch-graph-77ecc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:45.639641 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:45.639600    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" podUID="9d76e551-a8c7-43fa-81c0-a176fee9474a" containerName="switch-graph-77ecc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:48.684723 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:48.684656    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 24 21:26:50.639169 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:50.639128    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" podUID="9d76e551-a8c7-43fa-81c0-a176fee9474a" containerName="switch-graph-77ecc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:51.809623 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:51.809594    2522 generic.go:356] "Generic (PLEG): container finished" podID="9d76e551-a8c7-43fa-81c0-a176fee9474a" containerID="4ffdef2510b12910f3bc5ea331680df104f8d94419ab4c0ca1ff2429975017a4" exitCode=0
Jul 24 21:26:51.809922 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:51.809682    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" event={"ID":"9d76e551-a8c7-43fa-81c0-a176fee9474a","Type":"ContainerDied","Data":"4ffdef2510b12910f3bc5ea331680df104f8d94419ab4c0ca1ff2429975017a4"}
Jul 24 21:26:52.323336 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.323313    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:52.438283 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.438256    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d76e551-a8c7-43fa-81c0-a176fee9474a-openshift-service-ca-bundle\") pod \"9d76e551-a8c7-43fa-81c0-a176fee9474a\" (UID: \"9d76e551-a8c7-43fa-81c0-a176fee9474a\") "
Jul 24 21:26:52.438414 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.438334    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d76e551-a8c7-43fa-81c0-a176fee9474a-proxy-tls\") pod \"9d76e551-a8c7-43fa-81c0-a176fee9474a\" (UID: \"9d76e551-a8c7-43fa-81c0-a176fee9474a\") "
Jul 24 21:26:52.438663 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.438638    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d76e551-a8c7-43fa-81c0-a176fee9474a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9d76e551-a8c7-43fa-81c0-a176fee9474a" (UID: "9d76e551-a8c7-43fa-81c0-a176fee9474a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:26:52.440411 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.440390    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d76e551-a8c7-43fa-81c0-a176fee9474a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d76e551-a8c7-43fa-81c0-a176fee9474a" (UID: "9d76e551-a8c7-43fa-81c0-a176fee9474a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:26:52.539544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.539484    2522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d76e551-a8c7-43fa-81c0-a176fee9474a-openshift-service-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:26:52.539544 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.539509    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d76e551-a8c7-43fa-81c0-a176fee9474a-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:26:52.815433 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.815375    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"
Jul 24 21:26:52.815433 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.815404    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg" event={"ID":"9d76e551-a8c7-43fa-81c0-a176fee9474a","Type":"ContainerDied","Data":"886dfb9ad8522d783b0320dc846e73c9031e98d81e4897f2d9d8dab8c0868472"}
Jul 24 21:26:52.815839 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.815460    2522 scope.go:117] "RemoveContainer" containerID="4ffdef2510b12910f3bc5ea331680df104f8d94419ab4c0ca1ff2429975017a4"
Jul 24 21:26:52.835438 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.835415    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"]
Jul 24 21:26:52.840917 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:52.840901    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-77ecc-597566fc6b-4bnvg"]
Jul 24 21:26:54.407057 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:54.407021    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d76e551-a8c7-43fa-81c0-a176fee9474a" path="/var/lib/kubelet/pods/9d76e551-a8c7-43fa-81c0-a176fee9474a/volumes"
Jul 24 21:26:58.684948 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:26:58.684908    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 24 21:27:01.533148 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.533118    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"]
Jul 24 21:27:01.533615 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.533596    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d76e551-a8c7-43fa-81c0-a176fee9474a" containerName="switch-graph-77ecc"
Jul 24 21:27:01.533709 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.533619    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d76e551-a8c7-43fa-81c0-a176fee9474a" containerName="switch-graph-77ecc"
Jul 24 21:27:01.533709 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.533632    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kube-rbac-proxy"
Jul 24 21:27:01.533709 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.533639    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kube-rbac-proxy"
Jul 24 21:27:01.533709 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.533694    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kserve-container"
Jul 24 21:27:01.533709 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.533705    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kserve-container"
Jul 24 21:27:01.533949 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.533801    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kserve-container"
Jul 24 21:27:01.533949 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.533818    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d76e551-a8c7-43fa-81c0-a176fee9474a" containerName="switch-graph-77ecc"
Jul 24 21:27:01.533949 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.533831    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff6f332-3d37-402a-a5b4-d35b75b85475" containerName="kube-rbac-proxy"
Jul 24 21:27:01.538239 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.538220    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:01.538870 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.538833    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:01.540539 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.540517    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 24 21:27:01.540795 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.540779    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 24 21:27:01.546823 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.546802    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"]
Jul 24 21:27:01.604854 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.604832    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2376430-0dd9-49c4-9195-dd2871f56b3c-proxy-tls\") pod \"model-chainer-777d4d755d-5mb9s\" (UID: \"c2376430-0dd9-49c4-9195-dd2871f56b3c\") " pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:01.604978 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.604870    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2376430-0dd9-49c4-9195-dd2871f56b3c-openshift-service-ca-bundle\") pod \"model-chainer-777d4d755d-5mb9s\" (UID: \"c2376430-0dd9-49c4-9195-dd2871f56b3c\") " pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:01.705784 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.705756    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2376430-0dd9-49c4-9195-dd2871f56b3c-proxy-tls\") pod \"model-chainer-777d4d755d-5mb9s\" (UID: \"c2376430-0dd9-49c4-9195-dd2871f56b3c\") " pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:01.705926 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.705798    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2376430-0dd9-49c4-9195-dd2871f56b3c-openshift-service-ca-bundle\") pod \"model-chainer-777d4d755d-5mb9s\" (UID: \"c2376430-0dd9-49c4-9195-dd2871f56b3c\") " pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:01.705926 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:27:01.705913    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 24 21:27:01.706005 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:27:01.705998    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2376430-0dd9-49c4-9195-dd2871f56b3c-proxy-tls podName:c2376430-0dd9-49c4-9195-dd2871f56b3c nodeName:}" failed. No retries permitted until 2026-07-24 21:27:02.205975993 +0000 UTC m=+744.237370994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c2376430-0dd9-49c4-9195-dd2871f56b3c-proxy-tls") pod "model-chainer-777d4d755d-5mb9s" (UID: "c2376430-0dd9-49c4-9195-dd2871f56b3c") : secret "model-chainer-serving-cert" not found
Jul 24 21:27:01.706446 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:01.706429    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2376430-0dd9-49c4-9195-dd2871f56b3c-openshift-service-ca-bundle\") pod \"model-chainer-777d4d755d-5mb9s\" (UID: \"c2376430-0dd9-49c4-9195-dd2871f56b3c\") " pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:02.209820 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:02.209792    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2376430-0dd9-49c4-9195-dd2871f56b3c-proxy-tls\") pod \"model-chainer-777d4d755d-5mb9s\" (UID: \"c2376430-0dd9-49c4-9195-dd2871f56b3c\") " pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:02.212323 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:02.212299    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2376430-0dd9-49c4-9195-dd2871f56b3c-proxy-tls\") pod \"model-chainer-777d4d755d-5mb9s\" (UID: \"c2376430-0dd9-49c4-9195-dd2871f56b3c\") " pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:02.238121 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:02.238077    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:02.449598 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:02.449578    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:02.582265 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:02.582238    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"]
Jul 24 21:27:02.584491 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:27:02.584464    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2376430_0dd9_49c4_9195_dd2871f56b3c.slice/crio-56ca53661cc331c2d3e67a404b355a01c60a5a39d1ee001217784bb8cb00faed WatchSource:0}: Error finding container 56ca53661cc331c2d3e67a404b355a01c60a5a39d1ee001217784bb8cb00faed: Status 404 returned error can't find the container with id 56ca53661cc331c2d3e67a404b355a01c60a5a39d1ee001217784bb8cb00faed
Jul 24 21:27:02.861809 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:02.861732    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" event={"ID":"c2376430-0dd9-49c4-9195-dd2871f56b3c","Type":"ContainerStarted","Data":"bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d"}
Jul 24 21:27:02.861809 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:02.861783    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" event={"ID":"c2376430-0dd9-49c4-9195-dd2871f56b3c","Type":"ContainerStarted","Data":"56ca53661cc331c2d3e67a404b355a01c60a5a39d1ee001217784bb8cb00faed"}
Jul 24 21:27:02.861982 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:02.861881    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:02.882819 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:02.879283    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" podStartSLOduration=1.879265972 podStartE2EDuration="1.879265972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:27:02.877282029 +0000 UTC m=+744.908677036" watchObservedRunningTime="2026-07-24 21:27:02.879265972 +0000 UTC m=+744.910660981"
Jul 24 21:27:08.685362 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:08.685333    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:27:08.872382 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:08.872353    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:11.647054 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:11.647021    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"]
Jul 24 21:27:11.647439 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:11.647216    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" podUID="c2376430-0dd9-49c4-9195-dd2871f56b3c" containerName="model-chainer" containerID="cri-o://bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d" gracePeriod=30
Jul 24 21:27:11.670158 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:11.670136    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:11.783883 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:11.783842    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:11.877992 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:11.877964    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:11.947323 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:11.947294    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"]
Jul 24 21:27:11.950920 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:11.950899    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:11.953128 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:11.953109    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3da7-predictor-serving-cert\""
Jul 24 21:27:11.953523 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:11.953507    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3da7-kube-rbac-proxy-sar-config\""
Jul 24 21:27:11.962608 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:11.962583    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:11.962722 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:11.962695    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"]
Jul 24 21:27:12.080773 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.080748    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2vw6\" (UniqueName: \"kubernetes.io/projected/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-api-access-h2vw6\") pod \"error-404-isvc-c3da7-predictor-6d65f47599-b48qk\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") " pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:12.080887 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.080796    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dee7a60-efe2-4921-bd3d-dd373fb7b197-proxy-tls\") pod \"error-404-isvc-c3da7-predictor-6d65f47599-b48qk\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") " pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:12.080887 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.080821    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3da7-predictor-6d65f47599-b48qk\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") " pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:12.151956 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.151927    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:12.182052 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.182027    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2vw6\" (UniqueName: \"kubernetes.io/projected/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-api-access-h2vw6\") pod \"error-404-isvc-c3da7-predictor-6d65f47599-b48qk\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") " pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:12.182152 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.182073    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dee7a60-efe2-4921-bd3d-dd373fb7b197-proxy-tls\") pod \"error-404-isvc-c3da7-predictor-6d65f47599-b48qk\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") " pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:12.182152 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.182100    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3da7-predictor-6d65f47599-b48qk\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") " pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:12.182246 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:27:12.182190    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-serving-cert: secret "error-404-isvc-c3da7-predictor-serving-cert" not found
Jul 24 21:27:12.182297 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:27:12.182250    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dee7a60-efe2-4921-bd3d-dd373fb7b197-proxy-tls podName:9dee7a60-efe2-4921-bd3d-dd373fb7b197 nodeName:}" failed. No retries permitted until 2026-07-24 21:27:12.682229721 +0000 UTC m=+754.713624707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9dee7a60-efe2-4921-bd3d-dd373fb7b197-proxy-tls") pod "error-404-isvc-c3da7-predictor-6d65f47599-b48qk" (UID: "9dee7a60-efe2-4921-bd3d-dd373fb7b197") : secret "error-404-isvc-c3da7-predictor-serving-cert" not found
Jul 24 21:27:12.182786 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.182770    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3da7-predictor-6d65f47599-b48qk\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") " pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:12.190371 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.190349    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2vw6\" (UniqueName: \"kubernetes.io/projected/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-api-access-h2vw6\") pod \"error-404-isvc-c3da7-predictor-6d65f47599-b48qk\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") " pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:12.338856 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.338795    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:12.685419 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.685350    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dee7a60-efe2-4921-bd3d-dd373fb7b197-proxy-tls\") pod \"error-404-isvc-c3da7-predictor-6d65f47599-b48qk\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") " pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:12.687840 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.687822    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dee7a60-efe2-4921-bd3d-dd373fb7b197-proxy-tls\") pod \"error-404-isvc-c3da7-predictor-6d65f47599-b48qk\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") " pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:12.861127 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.861100    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:12.992696 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:12.992615    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"]
Jul 24 21:27:12.995021 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:27:12.994990    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dee7a60_efe2_4921_bd3d_dd373fb7b197.slice/crio-9c14fbecca0d090b21ea84ce49db8152210cdbef61a9065655bb382f347e38bb WatchSource:0}: Error finding container 9c14fbecca0d090b21ea84ce49db8152210cdbef61a9065655bb382f347e38bb: Status 404 returned error can't find the container with id 9c14fbecca0d090b21ea84ce49db8152210cdbef61a9065655bb382f347e38bb
Jul 24 21:27:13.870202 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:13.870161    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" podUID="c2376430-0dd9-49c4-9195-dd2871f56b3c" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:13.913357 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:13.913318    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" event={"ID":"9dee7a60-efe2-4921-bd3d-dd373fb7b197","Type":"ContainerStarted","Data":"c5fae28ade4474a7fbf52091c45f8fb79dc875780c8bd93b4f9c1d5260c79ce8"}
Jul 24 21:27:13.913508 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:13.913380    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" event={"ID":"9dee7a60-efe2-4921-bd3d-dd373fb7b197","Type":"ContainerStarted","Data":"617420a894fc9613ac497abc39de645bfae6403f8e7d3e92f502a789a73ab9a5"}
Jul 24 21:27:13.913508 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:13.913415    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:13.913508 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:13.913441    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" event={"ID":"9dee7a60-efe2-4921-bd3d-dd373fb7b197","Type":"ContainerStarted","Data":"9c14fbecca0d090b21ea84ce49db8152210cdbef61a9065655bb382f347e38bb"}
Jul 24 21:27:13.913508 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:13.913468    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:13.914538 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:13.914510    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 24 21:27:13.933237 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:13.933190    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" podStartSLOduration=2.9331736729999998 podStartE2EDuration="2.933173673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:27:13.930846823 +0000 UTC m=+755.962241831" watchObservedRunningTime="2026-07-24 21:27:13.933173673 +0000 UTC m=+755.964568681"
Jul 24 21:27:14.917836 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:14.917788    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 24 21:27:18.870895 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:18.870851    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" podUID="c2376430-0dd9-49c4-9195-dd2871f56b3c" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:19.921806 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:19.921780    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:27:19.922205 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:19.922179    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 24 21:27:21.925568 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:21.925530    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"]
Jul 24 21:27:21.929064 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:21.929042    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:27:21.929916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:21.929881    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:21.931584 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:21.931564    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-eea53-serving-cert\""
Jul 24 21:27:21.931697 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:21.931568    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-eea53-kube-rbac-proxy-sar-config\""
Jul 24 21:27:21.939854 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:21.939831    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"]
Jul 24 21:27:22.053705 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.053655    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7228bc66-4068-48e3-84ff-031be52336cb-proxy-tls\") pod \"switch-graph-eea53-7b4cdf96d5-xwlds\" (UID: \"7228bc66-4068-48e3-84ff-031be52336cb\") " pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:27:22.053856 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.053748    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7228bc66-4068-48e3-84ff-031be52336cb-openshift-service-ca-bundle\") pod \"switch-graph-eea53-7b4cdf96d5-xwlds\" (UID: \"7228bc66-4068-48e3-84ff-031be52336cb\") " pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:27:22.154590 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.154563    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7228bc66-4068-48e3-84ff-031be52336cb-proxy-tls\") pod \"switch-graph-eea53-7b4cdf96d5-xwlds\" (UID: \"7228bc66-4068-48e3-84ff-031be52336cb\") " pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:27:22.154758 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.154603    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7228bc66-4068-48e3-84ff-031be52336cb-openshift-service-ca-bundle\") pod \"switch-graph-eea53-7b4cdf96d5-xwlds\" (UID: \"7228bc66-4068-48e3-84ff-031be52336cb\") " pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:27:22.155191 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.155173    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7228bc66-4068-48e3-84ff-031be52336cb-openshift-service-ca-bundle\") pod \"switch-graph-eea53-7b4cdf96d5-xwlds\" (UID: \"7228bc66-4068-48e3-84ff-031be52336cb\") " pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:27:22.157041 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.157021    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7228bc66-4068-48e3-84ff-031be52336cb-proxy-tls\") pod \"switch-graph-eea53-7b4cdf96d5-xwlds\" (UID: \"7228bc66-4068-48e3-84ff-031be52336cb\") " pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:27:22.184584 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.184522    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:22.240645 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.240623    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:27:22.370469 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.370447    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"]
Jul 24 21:27:22.372843 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:27:22.372821    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7228bc66_4068_48e3_84ff_031be52336cb.slice/crio-333c6b4a44c92a4b207627705cccbb287ef9f10950585f754f8d3e12bb05b52f WatchSource:0}: Error finding container 333c6b4a44c92a4b207627705cccbb287ef9f10950585f754f8d3e12bb05b52f: Status 404 returned error can't find the container with id 333c6b4a44c92a4b207627705cccbb287ef9f10950585f754f8d3e12bb05b52f
Jul 24 21:27:22.954471 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.954440    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" event={"ID":"7228bc66-4068-48e3-84ff-031be52336cb","Type":"ContainerStarted","Data":"eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725"}
Jul 24 21:27:22.954852 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.954492    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" event={"ID":"7228bc66-4068-48e3-84ff-031be52336cb","Type":"ContainerStarted","Data":"333c6b4a44c92a4b207627705cccbb287ef9f10950585f754f8d3e12bb05b52f"}
Jul 24 21:27:22.954852 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.954557    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:27:22.971220 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:22.971170    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" podStartSLOduration=1.9711542789999998 podStartE2EDuration="1.971154279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:27:22.971089818 +0000 UTC m=+765.002484826" watchObservedRunningTime="2026-07-24 21:27:22.971154279 +0000 UTC m=+765.002549287"
Jul 24 21:27:23.870852 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:23.870820    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" podUID="c2376430-0dd9-49c4-9195-dd2871f56b3c" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:23.871023 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:23.870919    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:28.870281 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:28.870238    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" podUID="c2376430-0dd9-49c4-9195-dd2871f56b3c" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:28.965576 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:28.965545    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:27:29.922808 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:29.922773    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 24 21:27:33.869960 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:33.869916    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" podUID="c2376430-0dd9-49c4-9195-dd2871f56b3c" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:38.870737 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:38.870699    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" podUID="c2376430-0dd9-49c4-9195-dd2871f56b3c" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:39.922729 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:39.922664    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 24 21:27:41.807662 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:41.807638    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:41.903612 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:41.903551    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2376430-0dd9-49c4-9195-dd2871f56b3c-openshift-service-ca-bundle\") pod \"c2376430-0dd9-49c4-9195-dd2871f56b3c\" (UID: \"c2376430-0dd9-49c4-9195-dd2871f56b3c\") "
Jul 24 21:27:41.903750 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:41.903648    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2376430-0dd9-49c4-9195-dd2871f56b3c-proxy-tls\") pod \"c2376430-0dd9-49c4-9195-dd2871f56b3c\" (UID: \"c2376430-0dd9-49c4-9195-dd2871f56b3c\") "
Jul 24 21:27:41.903942 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:41.903924    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2376430-0dd9-49c4-9195-dd2871f56b3c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c2376430-0dd9-49c4-9195-dd2871f56b3c" (UID: "c2376430-0dd9-49c4-9195-dd2871f56b3c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:27:41.905646 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:41.905626    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2376430-0dd9-49c4-9195-dd2871f56b3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2376430-0dd9-49c4-9195-dd2871f56b3c" (UID: "c2376430-0dd9-49c4-9195-dd2871f56b3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:27:42.004876 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.004847    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2376430-0dd9-49c4-9195-dd2871f56b3c-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:27:42.004876 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.004873    2522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2376430-0dd9-49c4-9195-dd2871f56b3c-openshift-service-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:27:42.042205 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.042175    2522 generic.go:356] "Generic (PLEG): container finished" podID="c2376430-0dd9-49c4-9195-dd2871f56b3c" containerID="bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d" exitCode=0
Jul 24 21:27:42.042328 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.042219    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"
Jul 24 21:27:42.042328 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.042229    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" event={"ID":"c2376430-0dd9-49c4-9195-dd2871f56b3c","Type":"ContainerDied","Data":"bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d"}
Jul 24 21:27:42.042328 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.042285    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s" event={"ID":"c2376430-0dd9-49c4-9195-dd2871f56b3c","Type":"ContainerDied","Data":"56ca53661cc331c2d3e67a404b355a01c60a5a39d1ee001217784bb8cb00faed"}
Jul 24 21:27:42.042328 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.042318    2522 scope.go:117] "RemoveContainer" containerID="bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d"
Jul 24 21:27:42.051777 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.051761    2522 scope.go:117] "RemoveContainer" containerID="bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d"
Jul 24 21:27:42.052017 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:27:42.051999    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d\": container with ID starting with bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d not found: ID does not exist" containerID="bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d"
Jul 24 21:27:42.052064 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.052024    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d"} err="failed to get container status \"bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d\": rpc error: code = NotFound desc = could not find container \"bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d\": container with ID starting with bc669977573f22dcc78465476810d4f42ce8488f1f0b6e7b8fe7ae6a0eecbe9d not found: ID does not exist"
Jul 24 21:27:42.064142 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.064119    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"]
Jul 24 21:27:42.065603 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.065585    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-777d4d755d-5mb9s"]
Jul 24 21:27:42.407445 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:42.407416    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2376430-0dd9-49c4-9195-dd2871f56b3c" path="/var/lib/kubelet/pods/c2376430-0dd9-49c4-9195-dd2871f56b3c/volumes"
Jul 24 21:27:49.922220 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:49.922171    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 24 21:27:59.922853 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:27:59.922766    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:28:11.859373 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:11.859337    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"]
Jul 24 21:28:11.859874 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:11.859735    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2376430-0dd9-49c4-9195-dd2871f56b3c" containerName="model-chainer"
Jul 24 21:28:11.859874 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:11.859751    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2376430-0dd9-49c4-9195-dd2871f56b3c" containerName="model-chainer"
Jul 24 21:28:11.859874 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:11.859837    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2376430-0dd9-49c4-9195-dd2871f56b3c" containerName="model-chainer"
Jul 24 21:28:11.862811 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:11.862791    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:28:11.865034 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:11.865017    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c3da7-kube-rbac-proxy-sar-config\""
Jul 24 21:28:11.865122 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:11.865032    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c3da7-serving-cert\""
Jul 24 21:28:11.867411 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:11.867384    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:28:11.871403 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:11.871378    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"]
Jul 24 21:28:11.918210 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:11.918184    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-openshift-service-ca-bundle\") pod \"sequence-graph-c3da7-5dc8cb65f5-f9454\" (UID: \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\") " pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:28:11.918367 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:11.918225    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls\") pod \"sequence-graph-c3da7-5dc8cb65f5-f9454\" (UID: \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\") " pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:28:12.019082 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:12.019037    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls\") pod \"sequence-graph-c3da7-5dc8cb65f5-f9454\" (UID: \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\") " pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:28:12.019240 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:12.019130    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-openshift-service-ca-bundle\") pod \"sequence-graph-c3da7-5dc8cb65f5-f9454\" (UID: \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\") " pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:28:12.019240 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:28:12.019196    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-c3da7-serving-cert: secret "sequence-graph-c3da7-serving-cert" not found
Jul 24 21:28:12.019330 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:28:12.019272    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls podName:9ecafcfe-aedb-48b1-be80-6eabd1329a4d nodeName:}" failed. No retries permitted until 2026-07-24 21:28:12.519255639 +0000 UTC m=+814.550650625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls") pod "sequence-graph-c3da7-5dc8cb65f5-f9454" (UID: "9ecafcfe-aedb-48b1-be80-6eabd1329a4d") : secret "sequence-graph-c3da7-serving-cert" not found
Jul 24 21:28:12.019654 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:12.019638    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-openshift-service-ca-bundle\") pod \"sequence-graph-c3da7-5dc8cb65f5-f9454\" (UID: \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\") " pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:28:12.523338 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:12.523316    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls\") pod \"sequence-graph-c3da7-5dc8cb65f5-f9454\" (UID: \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\") " pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:28:12.523488 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:28:12.523433    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-c3da7-serving-cert: secret "sequence-graph-c3da7-serving-cert" not found
Jul 24 21:28:12.523530 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:28:12.523488    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls podName:9ecafcfe-aedb-48b1-be80-6eabd1329a4d nodeName:}" failed. No retries permitted until 2026-07-24 21:28:13.523475854 +0000 UTC m=+815.554870841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls") pod "sequence-graph-c3da7-5dc8cb65f5-f9454" (UID: "9ecafcfe-aedb-48b1-be80-6eabd1329a4d") : secret "sequence-graph-c3da7-serving-cert" not found
Jul 24 21:28:12.640739 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:12.640706    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:28:13.530549 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:13.530515    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls\") pod \"sequence-graph-c3da7-5dc8cb65f5-f9454\" (UID: \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\") " pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:28:13.533161 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:13.533139    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls\") pod \"sequence-graph-c3da7-5dc8cb65f5-f9454\" (UID: \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\") " pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:28:13.674365 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:13.674330    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:28:13.809849 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:13.809824    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"]
Jul 24 21:28:13.812219 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:28:13.812192    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecafcfe_aedb_48b1_be80_6eabd1329a4d.slice/crio-b864a22fea981c9ad24b44a4f8fbfe2c18e89a3d84aa7480e5f6cbb28be593be WatchSource:0}: Error finding container b864a22fea981c9ad24b44a4f8fbfe2c18e89a3d84aa7480e5f6cbb28be593be: Status 404 returned error can't find the container with id b864a22fea981c9ad24b44a4f8fbfe2c18e89a3d84aa7480e5f6cbb28be593be
Jul 24 21:28:14.193057 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:14.193026    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" event={"ID":"9ecafcfe-aedb-48b1-be80-6eabd1329a4d","Type":"ContainerStarted","Data":"3953a18875601197c3775986fa0d64b92c1d96847c0e70256b6a670170e517a4"}
Jul 24 21:28:14.193222 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:14.193082    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" event={"ID":"9ecafcfe-aedb-48b1-be80-6eabd1329a4d","Type":"ContainerStarted","Data":"b864a22fea981c9ad24b44a4f8fbfe2c18e89a3d84aa7480e5f6cbb28be593be"}
Jul 24 21:28:14.193222 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:14.193142    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:28:14.208661 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:14.208611    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" podStartSLOduration=3.208598744 podStartE2EDuration="3.208598744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:28:14.207643978 +0000 UTC m=+816.239038987" watchObservedRunningTime="2026-07-24 21:28:14.208598744 +0000 UTC m=+816.239993752"
Jul 24 21:28:20.205269 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:28:20.205230    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:29:41.190030 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:29:41.189934    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:30:37.170445 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:30:37.170393    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:12.147256 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:31:12.147152    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:49.762764 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:31:49.762713    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:32:10.566289 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:32:10.566253    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:36.623992 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.623906    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"]
Jul 24 21:35:36.624846 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.624148    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" podUID="7228bc66-4068-48e3-84ff-031be52336cb" containerName="switch-graph-eea53" containerID="cri-o://eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725" gracePeriod=30
Jul 24 21:35:36.672374 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.672345    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:36.773887 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.773856    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:36.853022 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.852992    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"]
Jul 24 21:35:36.853220 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.853194    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kserve-container" containerID="cri-o://9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8" gracePeriod=30
Jul 24 21:35:36.853294 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.853236    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kube-rbac-proxy" containerID="cri-o://af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944" gracePeriod=30
Jul 24 21:35:36.855664 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.855605    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:36.887022 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.886961    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"]
Jul 24 21:35:36.890612 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.890597    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:36.892686 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.892654    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b6aba-kube-rbac-proxy-sar-config\""
Jul 24 21:35:36.892835 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.892813    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b6aba-predictor-serving-cert\""
Jul 24 21:35:36.899930 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.899910    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"]
Jul 24 21:35:36.924950 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.924924    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:36.946905 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.946883    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b6aba-predictor-7b46948747-r5tnq\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") " pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:36.947064 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.946911    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-proxy-tls\") pod \"error-404-isvc-b6aba-predictor-7b46948747-r5tnq\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") " pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:36.947064 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:36.946933    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m77ql\" (UniqueName: \"kubernetes.io/projected/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-api-access-m77ql\") pod \"error-404-isvc-b6aba-predictor-7b46948747-r5tnq\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") " pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:37.047769 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.047739    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b6aba-predictor-7b46948747-r5tnq\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") " pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:37.047892 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.047779    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-proxy-tls\") pod \"error-404-isvc-b6aba-predictor-7b46948747-r5tnq\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") " pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:37.047892 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.047814    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m77ql\" (UniqueName: \"kubernetes.io/projected/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-api-access-m77ql\") pod \"error-404-isvc-b6aba-predictor-7b46948747-r5tnq\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") " pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:37.048005 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:35:37.047932    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-serving-cert: secret "error-404-isvc-b6aba-predictor-serving-cert" not found
Jul 24 21:35:37.048059 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:35:37.048021    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-proxy-tls podName:ed4d5fa6-13a7-4f14-9442-2e119333e8ae nodeName:}" failed. No retries permitted until 2026-07-24 21:35:37.547998506 +0000 UTC m=+1259.579393508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-proxy-tls") pod "error-404-isvc-b6aba-predictor-7b46948747-r5tnq" (UID: "ed4d5fa6-13a7-4f14-9442-2e119333e8ae") : secret "error-404-isvc-b6aba-predictor-serving-cert" not found
Jul 24 21:35:37.048570 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.048548    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b6aba-predictor-7b46948747-r5tnq\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") " pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:37.058214 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.058194    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m77ql\" (UniqueName: \"kubernetes.io/projected/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-api-access-m77ql\") pod \"error-404-isvc-b6aba-predictor-7b46948747-r5tnq\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") " pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:37.058648 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.058627    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:37.160201 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.160135    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:37.272345 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.272311    2522 generic.go:356] "Generic (PLEG): container finished" podID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerID="af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944" exitCode=2
Jul 24 21:35:37.272345 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.272334    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" event={"ID":"c798a1f6-74e8-48e5-ae62-fc1a546d596a","Type":"ContainerDied","Data":"af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944"}
Jul 24 21:35:37.552222 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.552180    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-proxy-tls\") pod \"error-404-isvc-b6aba-predictor-7b46948747-r5tnq\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") " pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:37.554735 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.554707    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-proxy-tls\") pod \"error-404-isvc-b6aba-predictor-7b46948747-r5tnq\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") " pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:37.801229 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.801185    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:37.934260 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:37.934235    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"]
Jul 24 21:35:37.936601 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:35:37.936570    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4d5fa6_13a7_4f14_9442_2e119333e8ae.slice/crio-6aa334ec33318ad1da672ec9ef35b88c2eb06359e74fc64f280d2d0c66a95799 WatchSource:0}: Error finding container 6aa334ec33318ad1da672ec9ef35b88c2eb06359e74fc64f280d2d0c66a95799: Status 404 returned error can't find the container with id 6aa334ec33318ad1da672ec9ef35b88c2eb06359e74fc64f280d2d0c66a95799
Jul 24 21:35:38.278775 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:38.278734    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" event={"ID":"ed4d5fa6-13a7-4f14-9442-2e119333e8ae","Type":"ContainerStarted","Data":"d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d"}
Jul 24 21:35:38.278924 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:38.278825    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" event={"ID":"ed4d5fa6-13a7-4f14-9442-2e119333e8ae","Type":"ContainerStarted","Data":"0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006"}
Jul 24 21:35:38.278924 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:38.278855    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:38.278924 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:38.278874    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" event={"ID":"ed4d5fa6-13a7-4f14-9442-2e119333e8ae","Type":"ContainerStarted","Data":"6aa334ec33318ad1da672ec9ef35b88c2eb06359e74fc64f280d2d0c66a95799"}
Jul 24 21:35:38.296998 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:38.296952    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" podStartSLOduration=2.296935551 podStartE2EDuration="2.296935551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:35:38.295800741 +0000 UTC m=+1260.327195799" watchObservedRunningTime="2026-07-24 21:35:38.296935551 +0000 UTC m=+1260.328330559"
Jul 24 21:35:38.679827 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:38.679731    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 24 21:35:38.684661 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:38.684627    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 24 21:35:38.964347 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:38.964306    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" podUID="7228bc66-4068-48e3-84ff-031be52336cb" containerName="switch-graph-eea53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:35:39.283989 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.283876    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:39.285150 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.285116    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 24 21:35:39.808978 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.808958    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:35:39.869781 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.869721    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-rbac-proxy-sar-config\") pod \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") "
Jul 24 21:35:39.869902 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.869791    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c798a1f6-74e8-48e5-ae62-fc1a546d596a-proxy-tls\") pod \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") "
Jul 24 21:35:39.869902 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.869821    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59rhx\" (UniqueName: \"kubernetes.io/projected/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-api-access-59rhx\") pod \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\" (UID: \"c798a1f6-74e8-48e5-ae62-fc1a546d596a\") "
Jul 24 21:35:39.870098 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.870073    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c798a1f6-74e8-48e5-ae62-fc1a546d596a" (UID: "c798a1f6-74e8-48e5-ae62-fc1a546d596a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:35:39.871885 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.871855    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c798a1f6-74e8-48e5-ae62-fc1a546d596a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c798a1f6-74e8-48e5-ae62-fc1a546d596a" (UID: "c798a1f6-74e8-48e5-ae62-fc1a546d596a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:35:39.871976 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.871889    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-api-access-59rhx" (OuterVolumeSpecName: "kube-api-access-59rhx") pod "c798a1f6-74e8-48e5-ae62-fc1a546d596a" (UID: "c798a1f6-74e8-48e5-ae62-fc1a546d596a"). InnerVolumeSpecName "kube-api-access-59rhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:35:39.970629 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.970603    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:35:39.970629 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.970627    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c798a1f6-74e8-48e5-ae62-fc1a546d596a-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:35:39.970991 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:39.970636    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59rhx\" (UniqueName: \"kubernetes.io/projected/c798a1f6-74e8-48e5-ae62-fc1a546d596a-kube-api-access-59rhx\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:35:40.290061 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.290030    2522 generic.go:356] "Generic (PLEG): container finished" podID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerID="9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8" exitCode=0
Jul 24 21:35:40.290212 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.290078    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"
Jul 24 21:35:40.290212 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.290112    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" event={"ID":"c798a1f6-74e8-48e5-ae62-fc1a546d596a","Type":"ContainerDied","Data":"9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8"}
Jul 24 21:35:40.290212 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.290183    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x" event={"ID":"c798a1f6-74e8-48e5-ae62-fc1a546d596a","Type":"ContainerDied","Data":"f8ec224bfa4812fdc0f6e12eb62441daca561cb4ba220784bd4fa9a95f7c53cf"}
Jul 24 21:35:40.290401 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.290221    2522 scope.go:117] "RemoveContainer" containerID="af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944"
Jul 24 21:35:40.290628 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.290598    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 24 21:35:40.299435 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.299420    2522 scope.go:117] "RemoveContainer" containerID="9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8"
Jul 24 21:35:40.306958 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.306935    2522 scope.go:117] "RemoveContainer" containerID="af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944"
Jul 24 21:35:40.307209 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:35:40.307181    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944\": container with ID starting with af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944 not found: ID does not exist" containerID="af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944"
Jul 24 21:35:40.307266 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.307210    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944"} err="failed to get container status \"af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944\": rpc error: code = NotFound desc = could not find container \"af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944\": container with ID starting with af3d5a010d2ae326336d36f3e050c81a9b7522a7ba065c1fde35327054e5a944 not found: ID does not exist"
Jul 24 21:35:40.307266 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.307226    2522 scope.go:117] "RemoveContainer" containerID="9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8"
Jul 24 21:35:40.307452 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:35:40.307435    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8\": container with ID starting with 9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8 not found: ID does not exist" containerID="9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8"
Jul 24 21:35:40.307485 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.307458    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8"} err="failed to get container status \"9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8\": rpc error: code = NotFound desc = could not find container \"9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8\": container with ID starting with 9475032f87d7d81fa6b5d2b8003105b610b5292a2deb39fc09410fbf4e320ff8 not found: ID does not exist"
Jul 24 21:35:40.310907 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.310885    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"]
Jul 24 21:35:40.315460 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.315442    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eea53-predictor-6bd495b56f-7z42x"]
Jul 24 21:35:40.406251 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:40.406228    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" path="/var/lib/kubelet/pods/c798a1f6-74e8-48e5-ae62-fc1a546d596a/volumes"
Jul 24 21:35:43.964313 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:43.964272    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" podUID="7228bc66-4068-48e3-84ff-031be52336cb" containerName="switch-graph-eea53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:35:45.295260 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:45.295229    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:35:45.295817 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:45.295790    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 24 21:35:48.964122 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:48.964086    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" podUID="7228bc66-4068-48e3-84ff-031be52336cb" containerName="switch-graph-eea53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:35:48.964573 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:48.964196    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:35:53.963402 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:53.963363    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" podUID="7228bc66-4068-48e3-84ff-031be52336cb" containerName="switch-graph-eea53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:35:55.295924 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:55.295885    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 24 21:35:58.963817 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:35:58.963771    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" podUID="7228bc66-4068-48e3-84ff-031be52336cb" containerName="switch-graph-eea53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:36:03.963717 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:03.963652    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" podUID="7228bc66-4068-48e3-84ff-031be52336cb" containerName="switch-graph-eea53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:36:05.296182 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:05.296133    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 24 21:36:06.775632 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:06.775604    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:36:06.867089 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:06.867065    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7228bc66-4068-48e3-84ff-031be52336cb-proxy-tls\") pod \"7228bc66-4068-48e3-84ff-031be52336cb\" (UID: \"7228bc66-4068-48e3-84ff-031be52336cb\") "
Jul 24 21:36:06.867244 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:06.867109    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7228bc66-4068-48e3-84ff-031be52336cb-openshift-service-ca-bundle\") pod \"7228bc66-4068-48e3-84ff-031be52336cb\" (UID: \"7228bc66-4068-48e3-84ff-031be52336cb\") "
Jul 24 21:36:06.867451 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:06.867429    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7228bc66-4068-48e3-84ff-031be52336cb-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7228bc66-4068-48e3-84ff-031be52336cb" (UID: "7228bc66-4068-48e3-84ff-031be52336cb"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:36:06.869276 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:06.869254    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7228bc66-4068-48e3-84ff-031be52336cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7228bc66-4068-48e3-84ff-031be52336cb" (UID: "7228bc66-4068-48e3-84ff-031be52336cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:36:06.968063 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:06.968041    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7228bc66-4068-48e3-84ff-031be52336cb-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:36:06.968063 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:06.968063    2522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7228bc66-4068-48e3-84ff-031be52336cb-openshift-service-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:36:07.432158 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:07.432084    2522 generic.go:356] "Generic (PLEG): container finished" podID="7228bc66-4068-48e3-84ff-031be52336cb" containerID="eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725" exitCode=0
Jul 24 21:36:07.432158 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:07.432126    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"
Jul 24 21:36:07.432328 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:07.432130    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" event={"ID":"7228bc66-4068-48e3-84ff-031be52336cb","Type":"ContainerDied","Data":"eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725"}
Jul 24 21:36:07.432328 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:07.432229    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds" event={"ID":"7228bc66-4068-48e3-84ff-031be52336cb","Type":"ContainerDied","Data":"333c6b4a44c92a4b207627705cccbb287ef9f10950585f754f8d3e12bb05b52f"}
Jul 24 21:36:07.432328 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:07.432252    2522 scope.go:117] "RemoveContainer" containerID="eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725"
Jul 24 21:36:07.441146 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:07.441130    2522 scope.go:117] "RemoveContainer" containerID="eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725"
Jul 24 21:36:07.441376 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:36:07.441357    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725\": container with ID starting with eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725 not found: ID does not exist" containerID="eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725"
Jul 24 21:36:07.441433 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:07.441384    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725"} err="failed to get container status \"eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725\": rpc error: code = NotFound desc = could not find container \"eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725\": container with ID starting with eb3687f9c97a1bf2c6c7115a6d1a459827913f8120159c5c0f658f962eddc725 not found: ID does not exist"
Jul 24 21:36:07.453315 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:07.453291    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"]
Jul 24 21:36:07.459840 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:07.459813    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-eea53-7b4cdf96d5-xwlds"]
Jul 24 21:36:08.407278 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:08.407249    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7228bc66-4068-48e3-84ff-031be52336cb" path="/var/lib/kubelet/pods/7228bc66-4068-48e3-84ff-031be52336cb/volumes"
Jul 24 21:36:15.295837 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:15.295796    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 24 21:36:25.296582 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:25.296558    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:36:26.538842 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:26.538814    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"]
Jul 24 21:36:26.539202 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:26.539008    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" podUID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" containerName="sequence-graph-c3da7" containerID="cri-o://3953a18875601197c3775986fa0d64b92c1d96847c0e70256b6a670170e517a4" gracePeriod=30
Jul 24 21:36:26.604262 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:26.604220    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:26.754026 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:26.753996    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"]
Jul 24 21:36:26.754237 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:26.754213    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kserve-container" containerID="cri-o://617420a894fc9613ac497abc39de645bfae6403f8e7d3e92f502a789a73ab9a5" gracePeriod=30
Jul 24 21:36:26.754299 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:26.754248    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kube-rbac-proxy" containerID="cri-o://c5fae28ade4474a7fbf52091c45f8fb79dc875780c8bd93b4f9c1d5260c79ce8" gracePeriod=30
Jul 24 21:36:26.768399 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:26.768374    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:26.904160 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:26.904129    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:27.027144 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.027087    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"]
Jul 24 21:36:27.027541 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.027518    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kube-rbac-proxy"
Jul 24 21:36:27.027541 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.027537    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kube-rbac-proxy"
Jul 24 21:36:27.027763 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.027556    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kserve-container"
Jul 24 21:36:27.027763 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.027561    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kserve-container"
Jul 24 21:36:27.027763 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.027577    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7228bc66-4068-48e3-84ff-031be52336cb" containerName="switch-graph-eea53"
Jul 24 21:36:27.027763 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.027583    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="7228bc66-4068-48e3-84ff-031be52336cb" containerName="switch-graph-eea53"
Jul 24 21:36:27.027763 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.027635    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kserve-container"
Jul 24 21:36:27.027763 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.027644    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="c798a1f6-74e8-48e5-ae62-fc1a546d596a" containerName="kube-rbac-proxy"
Jul 24 21:36:27.027763 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.027652    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="7228bc66-4068-48e3-84ff-031be52336cb" containerName="switch-graph-eea53"
Jul 24 21:36:27.030725 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.030708    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:27.033132 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.033113    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f761f-predictor-serving-cert\""
Jul 24 21:36:27.033448 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.033432    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f761f-kube-rbac-proxy-sar-config\""
Jul 24 21:36:27.044069 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.044037    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"]
Jul 24 21:36:27.063750 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.063713    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:27.117967 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.117936    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqt8n\" (UniqueName: \"kubernetes.io/projected/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-api-access-vqt8n\") pod \"error-404-isvc-f761f-predictor-54645fb6f5-cf88l\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") " pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:27.118117 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.117978    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f761f-predictor-54645fb6f5-cf88l\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") " pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:27.118117 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.118004    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fe3572-3832-4f45-bd9e-7124da1903cc-proxy-tls\") pod \"error-404-isvc-f761f-predictor-54645fb6f5-cf88l\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") " pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:27.218766 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.218740    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f761f-predictor-54645fb6f5-cf88l\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") " pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:27.218873 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.218775    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fe3572-3832-4f45-bd9e-7124da1903cc-proxy-tls\") pod \"error-404-isvc-f761f-predictor-54645fb6f5-cf88l\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") " pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:27.218873 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.218824    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqt8n\" (UniqueName: \"kubernetes.io/projected/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-api-access-vqt8n\") pod \"error-404-isvc-f761f-predictor-54645fb6f5-cf88l\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") " pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:27.218995 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:36:27.218975    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f761f-predictor-serving-cert: secret "error-404-isvc-f761f-predictor-serving-cert" not found
Jul 24 21:36:27.219042 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:36:27.219035    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5fe3572-3832-4f45-bd9e-7124da1903cc-proxy-tls podName:b5fe3572-3832-4f45-bd9e-7124da1903cc nodeName:}" failed. No retries permitted until 2026-07-24 21:36:27.719018888 +0000 UTC m=+1309.750413875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5fe3572-3832-4f45-bd9e-7124da1903cc-proxy-tls") pod "error-404-isvc-f761f-predictor-54645fb6f5-cf88l" (UID: "b5fe3572-3832-4f45-bd9e-7124da1903cc") : secret "error-404-isvc-f761f-predictor-serving-cert" not found
Jul 24 21:36:27.219395 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.219377    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f761f-predictor-54645fb6f5-cf88l\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") " pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:27.223004 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.222970    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:27.234504 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.234481    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqt8n\" (UniqueName: \"kubernetes.io/projected/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-api-access-vqt8n\") pod \"error-404-isvc-f761f-predictor-54645fb6f5-cf88l\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") " pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:27.486115 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.486036    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:27.524037 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.524002    2522 generic.go:356] "Generic (PLEG): container finished" podID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerID="c5fae28ade4474a7fbf52091c45f8fb79dc875780c8bd93b4f9c1d5260c79ce8" exitCode=2
Jul 24 21:36:27.524154 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.524076    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" event={"ID":"9dee7a60-efe2-4921-bd3d-dd373fb7b197","Type":"ContainerDied","Data":"c5fae28ade4474a7fbf52091c45f8fb79dc875780c8bd93b4f9c1d5260c79ce8"}
Jul 24 21:36:27.722570 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.722543    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fe3572-3832-4f45-bd9e-7124da1903cc-proxy-tls\") pod \"error-404-isvc-f761f-predictor-54645fb6f5-cf88l\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") " pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:27.724785 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.724763    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fe3572-3832-4f45-bd9e-7124da1903cc-proxy-tls\") pod \"error-404-isvc-f761f-predictor-54645fb6f5-cf88l\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") " pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:27.940885 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:27.940859    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:28.079943 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:28.079918    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"]
Jul 24 21:36:28.081846 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:36:28.081818    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fe3572_3832_4f45_bd9e_7124da1903cc.slice/crio-ca806bcbd68208509b1ca91c6e96f67ad7af68e5cde3908489e23ed06c91ff5f WatchSource:0}: Error finding container ca806bcbd68208509b1ca91c6e96f67ad7af68e5cde3908489e23ed06c91ff5f: Status 404 returned error can't find the container with id ca806bcbd68208509b1ca91c6e96f67ad7af68e5cde3908489e23ed06c91ff5f
Jul 24 21:36:28.529687 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:28.529584    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" event={"ID":"b5fe3572-3832-4f45-bd9e-7124da1903cc","Type":"ContainerStarted","Data":"9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7"}
Jul 24 21:36:28.529687 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:28.529648    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" event={"ID":"b5fe3572-3832-4f45-bd9e-7124da1903cc","Type":"ContainerStarted","Data":"09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a"}
Jul 24 21:36:28.529889 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:28.529709    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:28.529889 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:28.529732    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" event={"ID":"b5fe3572-3832-4f45-bd9e-7124da1903cc","Type":"ContainerStarted","Data":"ca806bcbd68208509b1ca91c6e96f67ad7af68e5cde3908489e23ed06c91ff5f"}
Jul 24 21:36:28.548119 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:28.548074    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" podStartSLOduration=2.548062026 podStartE2EDuration="2.548062026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:36:28.547111057 +0000 UTC m=+1310.578506087" watchObservedRunningTime="2026-07-24 21:36:28.548062026 +0000 UTC m=+1310.579457034"
Jul 24 21:36:29.544374 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.544341    2522 generic.go:356] "Generic (PLEG): container finished" podID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerID="617420a894fc9613ac497abc39de645bfae6403f8e7d3e92f502a789a73ab9a5" exitCode=0
Jul 24 21:36:29.544785 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.544412    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" event={"ID":"9dee7a60-efe2-4921-bd3d-dd373fb7b197","Type":"ContainerDied","Data":"617420a894fc9613ac497abc39de645bfae6403f8e7d3e92f502a789a73ab9a5"}
Jul 24 21:36:29.544785 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.544702    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:29.546006 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.545978    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 24 21:36:29.613724 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.613700    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:36:29.740878 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.740851    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2vw6\" (UniqueName: \"kubernetes.io/projected/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-api-access-h2vw6\") pod \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") "
Jul 24 21:36:29.741029 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.740905    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dee7a60-efe2-4921-bd3d-dd373fb7b197-proxy-tls\") pod \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") "
Jul 24 21:36:29.741029 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.741001    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-rbac-proxy-sar-config\") pod \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\" (UID: \"9dee7a60-efe2-4921-bd3d-dd373fb7b197\") "
Jul 24 21:36:29.741332 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.741310    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9dee7a60-efe2-4921-bd3d-dd373fb7b197" (UID: "9dee7a60-efe2-4921-bd3d-dd373fb7b197"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:36:29.743134 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.743111    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dee7a60-efe2-4921-bd3d-dd373fb7b197-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9dee7a60-efe2-4921-bd3d-dd373fb7b197" (UID: "9dee7a60-efe2-4921-bd3d-dd373fb7b197"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:36:29.743214 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.743144    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-api-access-h2vw6" (OuterVolumeSpecName: "kube-api-access-h2vw6") pod "9dee7a60-efe2-4921-bd3d-dd373fb7b197" (UID: "9dee7a60-efe2-4921-bd3d-dd373fb7b197"). InnerVolumeSpecName "kube-api-access-h2vw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:36:29.841648 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.841626    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dee7a60-efe2-4921-bd3d-dd373fb7b197-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:36:29.841648 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.841649    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:36:29.841800 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:29.841658    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2vw6\" (UniqueName: \"kubernetes.io/projected/9dee7a60-efe2-4921-bd3d-dd373fb7b197-kube-api-access-h2vw6\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:36:30.203911 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:30.203881    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" podUID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" containerName="sequence-graph-c3da7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:36:30.550754 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:30.550662    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"
Jul 24 21:36:30.550754 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:30.550704    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk" event={"ID":"9dee7a60-efe2-4921-bd3d-dd373fb7b197","Type":"ContainerDied","Data":"9c14fbecca0d090b21ea84ce49db8152210cdbef61a9065655bb382f347e38bb"}
Jul 24 21:36:30.551159 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:30.550769    2522 scope.go:117] "RemoveContainer" containerID="c5fae28ade4474a7fbf52091c45f8fb79dc875780c8bd93b4f9c1d5260c79ce8"
Jul 24 21:36:30.551219 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:30.551195    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 24 21:36:30.561345 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:30.561331    2522 scope.go:117] "RemoveContainer" containerID="617420a894fc9613ac497abc39de645bfae6403f8e7d3e92f502a789a73ab9a5"
Jul 24 21:36:30.577403 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:30.577384    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"]
Jul 24 21:36:30.585756 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:30.585735    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3da7-predictor-6d65f47599-b48qk"]
Jul 24 21:36:32.407504 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:32.407469    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" path="/var/lib/kubelet/pods/9dee7a60-efe2-4921-bd3d-dd373fb7b197/volumes"
Jul 24 21:36:35.203221 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:35.203188    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" podUID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" containerName="sequence-graph-c3da7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:36:35.555663 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:35.555586    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:36:35.556095 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:35.556065    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 24 21:36:36.883065 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.883032    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"]
Jul 24 21:36:36.883417 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.883368    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kube-rbac-proxy"
Jul 24 21:36:36.883417 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.883379    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kube-rbac-proxy"
Jul 24 21:36:36.883417 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.883395    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kserve-container"
Jul 24 21:36:36.883417 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.883401    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kserve-container"
Jul 24 21:36:36.883548 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.883455    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kube-rbac-proxy"
Jul 24 21:36:36.883548 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.883466    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dee7a60-efe2-4921-bd3d-dd373fb7b197" containerName="kserve-container"
Jul 24 21:36:36.886265 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.886250    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:36:36.888915 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.888893    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-b6aba-kube-rbac-proxy-sar-config\""
Jul 24 21:36:36.889038 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.888894    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-b6aba-serving-cert\""
Jul 24 21:36:36.905354 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.905334    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"]
Jul 24 21:36:36.920694 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.920638    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:36.999870 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.999848    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-openshift-service-ca-bundle\") pod \"ensemble-graph-b6aba-6bbf9ddf75-gt6b8\" (UID: \"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb\") " pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:36:36.999997 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:36.999888    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-proxy-tls\") pod \"ensemble-graph-b6aba-6bbf9ddf75-gt6b8\" (UID: \"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb\") " pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:36:37.054569 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:37.054537    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:37.100737 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:37.100707    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-openshift-service-ca-bundle\") pod \"ensemble-graph-b6aba-6bbf9ddf75-gt6b8\" (UID: \"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb\") " pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:36:37.100874 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:37.100753    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-proxy-tls\") pod \"ensemble-graph-b6aba-6bbf9ddf75-gt6b8\" (UID: \"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb\") " pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:36:37.101333 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:37.101312    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-openshift-service-ca-bundle\") pod \"ensemble-graph-b6aba-6bbf9ddf75-gt6b8\" (UID: \"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb\") " pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:36:37.103243 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:37.103222    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-proxy-tls\") pod \"ensemble-graph-b6aba-6bbf9ddf75-gt6b8\" (UID: \"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb\") " pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:36:37.197038 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:37.197018    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:36:37.353208 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:37.353141    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"]
Jul 24 21:36:37.356008 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:36:37.355977    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8b9fda_c439_4bc6_8ab6_0d59d683fefb.slice/crio-02bc24a4df644a85c7f9044f21d2e65f09ea0ab6e9350870712eb9b301e25213 WatchSource:0}: Error finding container 02bc24a4df644a85c7f9044f21d2e65f09ea0ab6e9350870712eb9b301e25213: Status 404 returned error can't find the container with id 02bc24a4df644a85c7f9044f21d2e65f09ea0ab6e9350870712eb9b301e25213
Jul 24 21:36:37.584153 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:37.584079    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" event={"ID":"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb","Type":"ContainerStarted","Data":"14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594"}
Jul 24 21:36:37.584153 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:37.584128    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" event={"ID":"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb","Type":"ContainerStarted","Data":"02bc24a4df644a85c7f9044f21d2e65f09ea0ab6e9350870712eb9b301e25213"}
Jul 24 21:36:37.584331 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:37.584202    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:36:37.600036 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:37.599992    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" podStartSLOduration=1.599977524 podStartE2EDuration="1.599977524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:36:37.599300144 +0000 UTC m=+1319.630695154" watchObservedRunningTime="2026-07-24 21:36:37.599977524 +0000 UTC m=+1319.631372531"
Jul 24 21:36:40.203338 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:40.203299    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" podUID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" containerName="sequence-graph-c3da7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:36:40.203829 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:40.203397    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:36:43.596864 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:43.596834    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:36:45.203121 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:45.203083    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" podUID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" containerName="sequence-graph-c3da7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:36:45.556308 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:45.556203    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 24 21:36:46.946519 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:46.946488    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"]
Jul 24 21:36:46.946949 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:46.946659    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" podUID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" containerName="ensemble-graph-b6aba" containerID="cri-o://14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594" gracePeriod=30
Jul 24 21:36:46.977010 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:46.976983    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:47.185748 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.185707    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:47.207838 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.207746    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"]
Jul 24 21:36:47.207984 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.207948    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kserve-container" containerID="cri-o://0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006" gracePeriod=30
Jul 24 21:36:47.208047 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.207991    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kube-rbac-proxy" containerID="cri-o://d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d" gracePeriod=30
Jul 24 21:36:47.280205 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.280183    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:47.326638 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.326614    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"]
Jul 24 21:36:47.329962 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.329946    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:47.333400 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.333380    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6de22-kube-rbac-proxy-sar-config\""
Jul 24 21:36:47.334024 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.334008    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6de22-predictor-serving-cert\""
Jul 24 21:36:47.346029 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.346012    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"]
Jul 24 21:36:47.379870 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.379844    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:47.471197 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.471132    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8782f4dd-7890-4128-aa45-087f880965a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6de22-predictor-6c4f986f4f-dx245\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") " pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:47.471197 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.471179    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2gp7\" (UniqueName: \"kubernetes.io/projected/8782f4dd-7890-4128-aa45-087f880965a6-kube-api-access-q2gp7\") pod \"error-404-isvc-6de22-predictor-6c4f986f4f-dx245\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") " pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:47.471337 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.471240    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8782f4dd-7890-4128-aa45-087f880965a6-proxy-tls\") pod \"error-404-isvc-6de22-predictor-6c4f986f4f-dx245\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") " pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:47.572467 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.572439    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8782f4dd-7890-4128-aa45-087f880965a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6de22-predictor-6c4f986f4f-dx245\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") " pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:47.572574 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.572482    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gp7\" (UniqueName: \"kubernetes.io/projected/8782f4dd-7890-4128-aa45-087f880965a6-kube-api-access-q2gp7\") pod \"error-404-isvc-6de22-predictor-6c4f986f4f-dx245\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") " pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:47.572574 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.572504    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8782f4dd-7890-4128-aa45-087f880965a6-proxy-tls\") pod \"error-404-isvc-6de22-predictor-6c4f986f4f-dx245\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") " pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:47.572665 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:36:47.572631    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-6de22-predictor-serving-cert: secret "error-404-isvc-6de22-predictor-serving-cert" not found
Jul 24 21:36:47.572747 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:36:47.572718    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8782f4dd-7890-4128-aa45-087f880965a6-proxy-tls podName:8782f4dd-7890-4128-aa45-087f880965a6 nodeName:}" failed. No retries permitted until 2026-07-24 21:36:48.072697676 +0000 UTC m=+1330.104092677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8782f4dd-7890-4128-aa45-087f880965a6-proxy-tls") pod "error-404-isvc-6de22-predictor-6c4f986f4f-dx245" (UID: "8782f4dd-7890-4128-aa45-087f880965a6") : secret "error-404-isvc-6de22-predictor-serving-cert" not found
Jul 24 21:36:47.573100 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.573082    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8782f4dd-7890-4128-aa45-087f880965a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6de22-predictor-6c4f986f4f-dx245\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") " pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:47.582954 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.582930    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gp7\" (UniqueName: \"kubernetes.io/projected/8782f4dd-7890-4128-aa45-087f880965a6-kube-api-access-q2gp7\") pod \"error-404-isvc-6de22-predictor-6c4f986f4f-dx245\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") " pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:47.633368 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.633343    2522 generic.go:356] "Generic (PLEG): container finished" podID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerID="d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d" exitCode=2
Jul 24 21:36:47.633508 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.633424    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" event={"ID":"ed4d5fa6-13a7-4f14-9442-2e119333e8ae","Type":"ContainerDied","Data":"d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d"}
Jul 24 21:36:47.820575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:47.820503    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:48.076428 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.076352    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8782f4dd-7890-4128-aa45-087f880965a6-proxy-tls\") pod \"error-404-isvc-6de22-predictor-6c4f986f4f-dx245\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") " pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:48.078867 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.078849    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8782f4dd-7890-4128-aa45-087f880965a6-proxy-tls\") pod \"error-404-isvc-6de22-predictor-6c4f986f4f-dx245\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") " pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:48.239424 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.239399    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:48.389018 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.388987    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"]
Jul 24 21:36:48.389990 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:36:48.389960    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8782f4dd_7890_4128_aa45_087f880965a6.slice/crio-35812cf7c83377dc99ec3fb019d189eb72e7bc210fea499a658bd053c05d8e0f WatchSource:0}: Error finding container 35812cf7c83377dc99ec3fb019d189eb72e7bc210fea499a658bd053c05d8e0f: Status 404 returned error can't find the container with id 35812cf7c83377dc99ec3fb019d189eb72e7bc210fea499a658bd053c05d8e0f
Jul 24 21:36:48.594758 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.594727    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" podUID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" containerName="ensemble-graph-b6aba" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:36:48.639441 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.639374    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" event={"ID":"8782f4dd-7890-4128-aa45-087f880965a6","Type":"ContainerStarted","Data":"afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6"}
Jul 24 21:36:48.639555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.639461    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" event={"ID":"8782f4dd-7890-4128-aa45-087f880965a6","Type":"ContainerStarted","Data":"61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376"}
Jul 24 21:36:48.639555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.639488    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:48.639555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.639516    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" event={"ID":"8782f4dd-7890-4128-aa45-087f880965a6","Type":"ContainerStarted","Data":"35812cf7c83377dc99ec3fb019d189eb72e7bc210fea499a658bd053c05d8e0f"}
Jul 24 21:36:48.639555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.639545    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:48.641101 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.641072    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 24 21:36:48.660065 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:48.660019    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" podStartSLOduration=1.660002568 podStartE2EDuration="1.660002568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:36:48.658825595 +0000 UTC m=+1330.690220608" watchObservedRunningTime="2026-07-24 21:36:48.660002568 +0000 UTC m=+1330.691397575"
Jul 24 21:36:49.643863 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:49.643827    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 24 21:36:50.204329 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.204290    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" podUID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" containerName="sequence-graph-c3da7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:36:50.263055 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.263035    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:36:50.396832 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.396760    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-rbac-proxy-sar-config\") pod \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") "
Jul 24 21:36:50.396832 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.396815    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-proxy-tls\") pod \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") "
Jul 24 21:36:50.397016 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.396878    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m77ql\" (UniqueName: \"kubernetes.io/projected/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-api-access-m77ql\") pod \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\" (UID: \"ed4d5fa6-13a7-4f14-9442-2e119333e8ae\") "
Jul 24 21:36:50.397091 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.397064    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed4d5fa6-13a7-4f14-9442-2e119333e8ae" (UID: "ed4d5fa6-13a7-4f14-9442-2e119333e8ae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:36:50.399048 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.399023    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-api-access-m77ql" (OuterVolumeSpecName: "kube-api-access-m77ql") pod "ed4d5fa6-13a7-4f14-9442-2e119333e8ae" (UID: "ed4d5fa6-13a7-4f14-9442-2e119333e8ae"). InnerVolumeSpecName "kube-api-access-m77ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:36:50.399157 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.399134    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed4d5fa6-13a7-4f14-9442-2e119333e8ae" (UID: "ed4d5fa6-13a7-4f14-9442-2e119333e8ae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:36:50.498318 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.498299    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:36:50.498318 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.498320    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:36:50.498449 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.498330    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m77ql\" (UniqueName: \"kubernetes.io/projected/ed4d5fa6-13a7-4f14-9442-2e119333e8ae-kube-api-access-m77ql\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:36:50.649764 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.649665    2522 generic.go:356] "Generic (PLEG): container finished" podID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerID="0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006" exitCode=0
Jul 24 21:36:50.649764 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.649707    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" event={"ID":"ed4d5fa6-13a7-4f14-9442-2e119333e8ae","Type":"ContainerDied","Data":"0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006"}
Jul 24 21:36:50.649764 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.649735    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"
Jul 24 21:36:50.650223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.649767    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq" event={"ID":"ed4d5fa6-13a7-4f14-9442-2e119333e8ae","Type":"ContainerDied","Data":"6aa334ec33318ad1da672ec9ef35b88c2eb06359e74fc64f280d2d0c66a95799"}
Jul 24 21:36:50.650223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.649795    2522 scope.go:117] "RemoveContainer" containerID="d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d"
Jul 24 21:36:50.658225 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.658203    2522 scope.go:117] "RemoveContainer" containerID="0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006"
Jul 24 21:36:50.665555 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.665540    2522 scope.go:117] "RemoveContainer" containerID="d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d"
Jul 24 21:36:50.665859 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:36:50.665841    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d\": container with ID starting with d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d not found: ID does not exist" containerID="d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d"
Jul 24 21:36:50.665916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.665868    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d"} err="failed to get container status \"d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d\": rpc error: code = NotFound desc = could not find container \"d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d\": container with ID starting with d089eea6ed783a9336427f23e31f52a4fbc89045d5128dd9a325e47b5ef3077d not found: ID does not exist"
Jul 24 21:36:50.665916 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.665884    2522 scope.go:117] "RemoveContainer" containerID="0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006"
Jul 24 21:36:50.666123 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:36:50.666105    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006\": container with ID starting with 0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006 not found: ID does not exist" containerID="0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006"
Jul 24 21:36:50.666168 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.666129    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006"} err="failed to get container status \"0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006\": rpc error: code = NotFound desc = could not find container \"0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006\": container with ID starting with 0084bfeafa333dc6c45b5866ee24b3030d701a8338b71a1d5a93dff8fa5ee006 not found: ID does not exist"
Jul 24 21:36:50.673165 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.673143    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"]
Jul 24 21:36:50.678208 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:50.678190    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b6aba-predictor-7b46948747-r5tnq"]
Jul 24 21:36:52.407206 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:52.407176    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" path="/var/lib/kubelet/pods/ed4d5fa6-13a7-4f14-9442-2e119333e8ae/volumes"
Jul 24 21:36:53.594171 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:53.594131    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" podUID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" containerName="ensemble-graph-b6aba" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:36:54.648280 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:54.648251    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:36:54.648826 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:54.648800    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 24 21:36:55.203284 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:55.203250    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" podUID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" containerName="sequence-graph-c3da7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:36:55.556794 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:55.556715    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 24 21:36:56.679927 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:56.679896    2522 generic.go:356] "Generic (PLEG): container finished" podID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" containerID="3953a18875601197c3775986fa0d64b92c1d96847c0e70256b6a670170e517a4" exitCode=0
Jul 24 21:36:56.680255 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:56.679964    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" event={"ID":"9ecafcfe-aedb-48b1-be80-6eabd1329a4d","Type":"ContainerDied","Data":"3953a18875601197c3775986fa0d64b92c1d96847c0e70256b6a670170e517a4"}
Jul 24 21:36:56.680255 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:56.680028    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454" event={"ID":"9ecafcfe-aedb-48b1-be80-6eabd1329a4d","Type":"ContainerDied","Data":"b864a22fea981c9ad24b44a4f8fbfe2c18e89a3d84aa7480e5f6cbb28be593be"}
Jul 24 21:36:56.680255 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:56.680055    2522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b864a22fea981c9ad24b44a4f8fbfe2c18e89a3d84aa7480e5f6cbb28be593be"
Jul 24 21:36:56.688238 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:56.688221    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:36:56.742234 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:56.742217    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-openshift-service-ca-bundle\") pod \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\" (UID: \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\") "
Jul 24 21:36:56.742400 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:56.742261    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls\") pod \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\" (UID: \"9ecafcfe-aedb-48b1-be80-6eabd1329a4d\") "
Jul 24 21:36:56.742616 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:56.742594    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9ecafcfe-aedb-48b1-be80-6eabd1329a4d" (UID: "9ecafcfe-aedb-48b1-be80-6eabd1329a4d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:36:56.744330 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:56.744307    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ecafcfe-aedb-48b1-be80-6eabd1329a4d" (UID: "9ecafcfe-aedb-48b1-be80-6eabd1329a4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:36:56.843229 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:56.843159    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:36:56.843229 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:56.843181    2522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ecafcfe-aedb-48b1-be80-6eabd1329a4d-openshift-service-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:36:57.684392 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:57.684361    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"
Jul 24 21:36:57.706660 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:57.706635    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"]
Jul 24 21:36:57.710779 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:57.710757    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c3da7-5dc8cb65f5-f9454"]
Jul 24 21:36:58.406872 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:58.406838    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" path="/var/lib/kubelet/pods/9ecafcfe-aedb-48b1-be80-6eabd1329a4d/volumes"
Jul 24 21:36:58.594062 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:58.594027    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" podUID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" containerName="ensemble-graph-b6aba" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:36:58.594226 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:36:58.594119    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:37:03.594185 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:03.594094    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" podUID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" containerName="ensemble-graph-b6aba" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:37:04.648899 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:04.648864    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 24 21:37:05.556892 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:05.556856    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 24 21:37:08.593597 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:08.593555    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" podUID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" containerName="ensemble-graph-b6aba" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:37:13.593733 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:13.593696    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" podUID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" containerName="ensemble-graph-b6aba" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:37:14.649141 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:14.649103    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 24 21:37:15.557242 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:15.557213    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:37:17.099009 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.098988    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:37:17.194896 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.194875    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-proxy-tls\") pod \"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb\" (UID: \"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb\") "
Jul 24 21:37:17.195020 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.194944    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-openshift-service-ca-bundle\") pod \"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb\" (UID: \"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb\") "
Jul 24 21:37:17.195287 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.195267    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" (UID: "9d8b9fda-c439-4bc6-8ab6-0d59d683fefb"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:37:17.196907 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.196889    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" (UID: "9d8b9fda-c439-4bc6-8ab6-0d59d683fefb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:37:17.295518 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.295498    2522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-openshift-service-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:37:17.295518 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.295518    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:37:17.774908 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.774878    2522 generic.go:356] "Generic (PLEG): container finished" podID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" containerID="14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594" exitCode=0
Jul 24 21:37:17.775081 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.774927    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"
Jul 24 21:37:17.775081 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.774971    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" event={"ID":"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb","Type":"ContainerDied","Data":"14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594"}
Jul 24 21:37:17.775081 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.775032    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8" event={"ID":"9d8b9fda-c439-4bc6-8ab6-0d59d683fefb","Type":"ContainerDied","Data":"02bc24a4df644a85c7f9044f21d2e65f09ea0ab6e9350870712eb9b301e25213"}
Jul 24 21:37:17.775081 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.775061    2522 scope.go:117] "RemoveContainer" containerID="14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594"
Jul 24 21:37:17.784088 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.784063    2522 scope.go:117] "RemoveContainer" containerID="14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594"
Jul 24 21:37:17.784382 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:37:17.784364    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594\": container with ID starting with 14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594 not found: ID does not exist" containerID="14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594"
Jul 24 21:37:17.784424 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.784392    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594"} err="failed to get container status \"14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594\": rpc error: code = NotFound desc = could not find container \"14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594\": container with ID starting with 14c6e0bbc925a154268303b6e89ce7d15a858d40a98ecf01ac7a7f2131e01594 not found: ID does not exist"
Jul 24 21:37:17.797014 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.796981    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"]
Jul 24 21:37:17.801661 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:17.801641    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b6aba-6bbf9ddf75-gt6b8"]
Jul 24 21:37:18.406976 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:18.406952    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" path="/var/lib/kubelet/pods/9d8b9fda-c439-4bc6-8ab6-0d59d683fefb/volumes"
Jul 24 21:37:24.648811 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:24.648773    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 24 21:37:26.808012 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.807969    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"]
Jul 24 21:37:26.808486 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808467    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kube-rbac-proxy"
Jul 24 21:37:26.808575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808490    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kube-rbac-proxy"
Jul 24 21:37:26.808575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808512    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" containerName="ensemble-graph-b6aba"
Jul 24 21:37:26.808575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808521    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" containerName="ensemble-graph-b6aba"
Jul 24 21:37:26.808575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808537    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" containerName="sequence-graph-c3da7"
Jul 24 21:37:26.808575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808545    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" containerName="sequence-graph-c3da7"
Jul 24 21:37:26.808575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808565    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kserve-container"
Jul 24 21:37:26.808575 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808573    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kserve-container"
Jul 24 21:37:26.808942 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808659    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8b9fda-c439-4bc6-8ab6-0d59d683fefb" containerName="ensemble-graph-b6aba"
Jul 24 21:37:26.808942 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808690    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ecafcfe-aedb-48b1-be80-6eabd1329a4d" containerName="sequence-graph-c3da7"
Jul 24 21:37:26.808942 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808701    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kserve-container"
Jul 24 21:37:26.808942 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.808710    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4d5fa6-13a7-4f14-9442-2e119333e8ae" containerName="kube-rbac-proxy"
Jul 24 21:37:26.813072 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.813026    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:37:26.815538 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.815509    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f761f-kube-rbac-proxy-sar-config\""
Jul 24 21:37:26.815659 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.815511    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f761f-serving-cert\""
Jul 24 21:37:26.822216 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.822185    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"]
Jul 24 21:37:26.827859 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.827834    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:26.858243 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.858215    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e149d103-691c-4e5c-ac97-f5051ec0dea5-openshift-service-ca-bundle\") pod \"sequence-graph-f761f-8d6759985-j4lvt\" (UID: \"e149d103-691c-4e5c-ac97-f5051ec0dea5\") " pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:37:26.858345 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.858246    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e149d103-691c-4e5c-ac97-f5051ec0dea5-proxy-tls\") pod \"sequence-graph-f761f-8d6759985-j4lvt\" (UID: \"e149d103-691c-4e5c-ac97-f5051ec0dea5\") " pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:37:26.958646 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.958612    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e149d103-691c-4e5c-ac97-f5051ec0dea5-openshift-service-ca-bundle\") pod \"sequence-graph-f761f-8d6759985-j4lvt\" (UID: \"e149d103-691c-4e5c-ac97-f5051ec0dea5\") " pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:37:26.958646 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.958649    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e149d103-691c-4e5c-ac97-f5051ec0dea5-proxy-tls\") pod \"sequence-graph-f761f-8d6759985-j4lvt\" (UID: \"e149d103-691c-4e5c-ac97-f5051ec0dea5\") " pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:37:26.959256 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.959237    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e149d103-691c-4e5c-ac97-f5051ec0dea5-openshift-service-ca-bundle\") pod \"sequence-graph-f761f-8d6759985-j4lvt\" (UID: \"e149d103-691c-4e5c-ac97-f5051ec0dea5\") " pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:37:26.961061 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:26.961040    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e149d103-691c-4e5c-ac97-f5051ec0dea5-proxy-tls\") pod \"sequence-graph-f761f-8d6759985-j4lvt\" (UID: \"e149d103-691c-4e5c-ac97-f5051ec0dea5\") " pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:37:27.130183 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:27.130119    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:37:27.270006 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:27.269984    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"]
Jul 24 21:37:27.271964 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:37:27.271938    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode149d103_691c_4e5c_ac97_f5051ec0dea5.slice/crio-731086bf0425992995d37d6e5302dc924b6c2f1d7321f5139091039070db429c WatchSource:0}: Error finding container 731086bf0425992995d37d6e5302dc924b6c2f1d7321f5139091039070db429c: Status 404 returned error can't find the container with id 731086bf0425992995d37d6e5302dc924b6c2f1d7321f5139091039070db429c
Jul 24 21:37:27.824846 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:27.824811    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" event={"ID":"e149d103-691c-4e5c-ac97-f5051ec0dea5","Type":"ContainerStarted","Data":"5aba7874920c7f63dbb1970ff406fb16e1e8763ca869cf45dc54ccb8809e5f53"}
Jul 24 21:37:27.825215 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:27.824864    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" event={"ID":"e149d103-691c-4e5c-ac97-f5051ec0dea5","Type":"ContainerStarted","Data":"731086bf0425992995d37d6e5302dc924b6c2f1d7321f5139091039070db429c"}
Jul 24 21:37:27.825215 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:27.824884    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:37:27.844279 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:27.844237    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" podStartSLOduration=1.844221481 podStartE2EDuration="1.844221481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:37:27.843356365 +0000 UTC m=+1369.874751396" watchObservedRunningTime="2026-07-24 21:37:27.844221481 +0000 UTC m=+1369.875616489"
Jul 24 21:37:28.111291 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:28.111216    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:33.834463 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:33.834432    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:37:34.649525 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:34.649497    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:37:36.895366 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:36.895334    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"]
Jul 24 21:37:36.895824 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:36.895570    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" podUID="e149d103-691c-4e5c-ac97-f5051ec0dea5" containerName="sequence-graph-f761f" containerID="cri-o://5aba7874920c7f63dbb1970ff406fb16e1e8763ca869cf45dc54ccb8809e5f53" gracePeriod=30
Jul 24 21:37:36.917086 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:36.917060    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:37.051594 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.051560    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:37.083799 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.083772    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"]
Jul 24 21:37:37.084086 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.084054    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kserve-container" containerID="cri-o://09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a" gracePeriod=30
Jul 24 21:37:37.084086 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.084054    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kube-rbac-proxy" containerID="cri-o://9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7" gracePeriod=30
Jul 24 21:37:37.142102 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.142073    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:37.212015 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.211991    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"]
Jul 24 21:37:37.215259 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.215242    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.217426 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.217405    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-59fd1-kube-rbac-proxy-sar-config\""
Jul 24 21:37:37.217640 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.217628    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-59fd1-predictor-serving-cert\""
Jul 24 21:37:37.227294 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.227273    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"]
Jul 24 21:37:37.235776 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.235754    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:37.295823 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.295797    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:37.336070 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.336045    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bqnh\" (UniqueName: \"kubernetes.io/projected/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-api-access-9bqnh\") pod \"error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") " pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.336190 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.336110    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21056da-69b4-4ad2-bd14-a65e99a848ed-proxy-tls\") pod \"error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") " pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.336190 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.336175    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") " pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.436932 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.436906    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bqnh\" (UniqueName: \"kubernetes.io/projected/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-api-access-9bqnh\") pod \"error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") " pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.437026 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.436944    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21056da-69b4-4ad2-bd14-a65e99a848ed-proxy-tls\") pod \"error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") " pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.437026 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.436964    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") " pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.437603 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.437578    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") " pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.439422 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.439400    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21056da-69b4-4ad2-bd14-a65e99a848ed-proxy-tls\") pod \"error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") " pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.444751 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.444728    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bqnh\" (UniqueName: \"kubernetes.io/projected/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-api-access-9bqnh\") pod \"error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") " pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.525430 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.525383    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.658966 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.658902    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"]
Jul 24 21:37:37.661134 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:37:37.661111    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21056da_69b4_4ad2_bd14_a65e99a848ed.slice/crio-36eec655de851c36d451fa292a015fc17b10442650f7f145adaa2f1555ba1638 WatchSource:0}: Error finding container 36eec655de851c36d451fa292a015fc17b10442650f7f145adaa2f1555ba1638: Status 404 returned error can't find the container with id 36eec655de851c36d451fa292a015fc17b10442650f7f145adaa2f1555ba1638
Jul 24 21:37:37.873338 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.873263    2522 generic.go:356] "Generic (PLEG): container finished" podID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerID="9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7" exitCode=2
Jul 24 21:37:37.873465 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.873333    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" event={"ID":"b5fe3572-3832-4f45-bd9e-7124da1903cc","Type":"ContainerDied","Data":"9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7"}
Jul 24 21:37:37.875036 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.875015    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" event={"ID":"c21056da-69b4-4ad2-bd14-a65e99a848ed","Type":"ContainerStarted","Data":"58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db"}
Jul 24 21:37:37.875128 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.875068    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" event={"ID":"c21056da-69b4-4ad2-bd14-a65e99a848ed","Type":"ContainerStarted","Data":"348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca"}
Jul 24 21:37:37.875128 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.875097    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" event={"ID":"c21056da-69b4-4ad2-bd14-a65e99a848ed","Type":"ContainerStarted","Data":"36eec655de851c36d451fa292a015fc17b10442650f7f145adaa2f1555ba1638"}
Jul 24 21:37:37.875223 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.875209    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:37.902809 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:37.902763    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" podStartSLOduration=0.902749248 podStartE2EDuration="902.749248ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:37:37.901656744 +0000 UTC m=+1379.933051752" watchObservedRunningTime="2026-07-24 21:37:37.902749248 +0000 UTC m=+1379.934144257"
Jul 24 21:37:38.465283 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:38.465259    2522 scope.go:117] "RemoveContainer" containerID="3953a18875601197c3775986fa0d64b92c1d96847c0e70256b6a670170e517a4"
Jul 24 21:37:38.833356 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:38.833268    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" podUID="e149d103-691c-4e5c-ac97-f5051ec0dea5" containerName="sequence-graph-f761f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:37:38.880824 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:38.880787    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:38.882255 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:38.882220    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 24 21:37:39.884580 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:39.884544    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 24 21:37:40.035773 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.035752    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:37:40.158763 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.158698    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-rbac-proxy-sar-config\") pod \"b5fe3572-3832-4f45-bd9e-7124da1903cc\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") "
Jul 24 21:37:40.158892 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.158775    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqt8n\" (UniqueName: \"kubernetes.io/projected/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-api-access-vqt8n\") pod \"b5fe3572-3832-4f45-bd9e-7124da1903cc\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") "
Jul 24 21:37:40.158892 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.158811    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fe3572-3832-4f45-bd9e-7124da1903cc-proxy-tls\") pod \"b5fe3572-3832-4f45-bd9e-7124da1903cc\" (UID: \"b5fe3572-3832-4f45-bd9e-7124da1903cc\") "
Jul 24 21:37:40.159019 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.158997    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5fe3572-3832-4f45-bd9e-7124da1903cc" (UID: "b5fe3572-3832-4f45-bd9e-7124da1903cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:37:40.161002 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.160977    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5fe3572-3832-4f45-bd9e-7124da1903cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5fe3572-3832-4f45-bd9e-7124da1903cc" (UID: "b5fe3572-3832-4f45-bd9e-7124da1903cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:37:40.161103 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.161013    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-api-access-vqt8n" (OuterVolumeSpecName: "kube-api-access-vqt8n") pod "b5fe3572-3832-4f45-bd9e-7124da1903cc" (UID: "b5fe3572-3832-4f45-bd9e-7124da1903cc"). InnerVolumeSpecName "kube-api-access-vqt8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:37:40.260236 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.260204    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:37:40.260236 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.260235    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqt8n\" (UniqueName: \"kubernetes.io/projected/b5fe3572-3832-4f45-bd9e-7124da1903cc-kube-api-access-vqt8n\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:37:40.260236 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.260244    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5fe3572-3832-4f45-bd9e-7124da1903cc-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:37:40.890602 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.890569    2522 generic.go:356] "Generic (PLEG): container finished" podID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerID="09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a" exitCode=0
Jul 24 21:37:40.891026 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.890623    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"
Jul 24 21:37:40.891026 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.890666    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" event={"ID":"b5fe3572-3832-4f45-bd9e-7124da1903cc","Type":"ContainerDied","Data":"09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a"}
Jul 24 21:37:40.891026 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.890791    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l" event={"ID":"b5fe3572-3832-4f45-bd9e-7124da1903cc","Type":"ContainerDied","Data":"ca806bcbd68208509b1ca91c6e96f67ad7af68e5cde3908489e23ed06c91ff5f"}
Jul 24 21:37:40.891026 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.890835    2522 scope.go:117] "RemoveContainer" containerID="9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7"
Jul 24 21:37:40.899457 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.899434    2522 scope.go:117] "RemoveContainer" containerID="09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a"
Jul 24 21:37:40.906779 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.906583    2522 scope.go:117] "RemoveContainer" containerID="9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7"
Jul 24 21:37:40.906880 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:37:40.906861    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7\": container with ID starting with 9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7 not found: ID does not exist" containerID="9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7"
Jul 24 21:37:40.906926 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.906887    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7"} err="failed to get container status \"9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7\": rpc error: code = NotFound desc = could not find container \"9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7\": container with ID starting with 9ad0d05fed33c65f9bd516e9801852e694819e71328ca938a7f0773ed1855ce7 not found: ID does not exist"
Jul 24 21:37:40.906926 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.906904    2522 scope.go:117] "RemoveContainer" containerID="09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a"
Jul 24 21:37:40.907165 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:37:40.907134    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a\": container with ID starting with 09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a not found: ID does not exist" containerID="09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a"
Jul 24 21:37:40.907209 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.907169    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a"} err="failed to get container status \"09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a\": rpc error: code = NotFound desc = could not find container \"09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a\": container with ID starting with 09492afdbe883fd3ab99da413f0f0e7ac17cdc86d27bc7b60b0870180d7fd02a not found: ID does not exist"
Jul 24 21:37:40.910362 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.910334    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"]
Jul 24 21:37:40.914760 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:40.914735    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f761f-predictor-54645fb6f5-cf88l"]
Jul 24 21:37:42.406973 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:42.406937    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" path="/var/lib/kubelet/pods/b5fe3572-3832-4f45-bd9e-7124da1903cc/volumes"
Jul 24 21:37:43.833198 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:43.833160    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" podUID="e149d103-691c-4e5c-ac97-f5051ec0dea5" containerName="sequence-graph-f761f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:37:44.889269 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:44.889240    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:37:44.889885 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:44.889859    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 24 21:37:47.214476 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.214440    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"]
Jul 24 21:37:47.214914 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.214829    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kserve-container"
Jul 24 21:37:47.214914 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.214848    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kserve-container"
Jul 24 21:37:47.214914 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.214864    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kube-rbac-proxy"
Jul 24 21:37:47.214914 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.214873    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kube-rbac-proxy"
Jul 24 21:37:47.215112 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.214971    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kube-rbac-proxy"
Jul 24 21:37:47.215112 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.214980    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5fe3572-3832-4f45-bd9e-7124da1903cc" containerName="kserve-container"
Jul 24 21:37:47.218993 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.218943    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:47.219662 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.219649    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:37:47.221884 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.221864    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-6de22-kube-rbac-proxy-sar-config\""
Jul 24 21:37:47.222634 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.222613    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-6de22-serving-cert\""
Jul 24 21:37:47.228758 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.228738    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"]
Jul 24 21:37:47.313027 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.312999    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b06916f-f4be-4bee-b908-7e7ca985c474-openshift-service-ca-bundle\") pod \"ensemble-graph-6de22-7f888f584b-85sxv\" (UID: \"4b06916f-f4be-4bee-b908-7e7ca985c474\") " pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:37:47.313179 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.313041    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b06916f-f4be-4bee-b908-7e7ca985c474-proxy-tls\") pod \"ensemble-graph-6de22-7f888f584b-85sxv\" (UID: \"4b06916f-f4be-4bee-b908-7e7ca985c474\") " pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:37:47.413492 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.413465    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b06916f-f4be-4bee-b908-7e7ca985c474-openshift-service-ca-bundle\") pod \"ensemble-graph-6de22-7f888f584b-85sxv\" (UID: \"4b06916f-f4be-4bee-b908-7e7ca985c474\") " pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:37:47.413649 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.413500    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b06916f-f4be-4bee-b908-7e7ca985c474-proxy-tls\") pod \"ensemble-graph-6de22-7f888f584b-85sxv\" (UID: \"4b06916f-f4be-4bee-b908-7e7ca985c474\") " pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:37:47.413649 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:37:47.413610    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-6de22-serving-cert: secret "ensemble-graph-6de22-serving-cert" not found
Jul 24 21:37:47.413806 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:37:47.413697    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b06916f-f4be-4bee-b908-7e7ca985c474-proxy-tls podName:4b06916f-f4be-4bee-b908-7e7ca985c474 nodeName:}" failed. No retries permitted until 2026-07-24 21:37:47.913652943 +0000 UTC m=+1389.945047928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4b06916f-f4be-4bee-b908-7e7ca985c474-proxy-tls") pod "ensemble-graph-6de22-7f888f584b-85sxv" (UID: "4b06916f-f4be-4bee-b908-7e7ca985c474") : secret "ensemble-graph-6de22-serving-cert" not found
Jul 24 21:37:47.414157 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.414134    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b06916f-f4be-4bee-b908-7e7ca985c474-openshift-service-ca-bundle\") pod \"ensemble-graph-6de22-7f888f584b-85sxv\" (UID: \"4b06916f-f4be-4bee-b908-7e7ca985c474\") " pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:37:47.665376 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.665284    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:47.917896 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.917813    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b06916f-f4be-4bee-b908-7e7ca985c474-proxy-tls\") pod \"ensemble-graph-6de22-7f888f584b-85sxv\" (UID: \"4b06916f-f4be-4bee-b908-7e7ca985c474\") " pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:37:47.920329 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:47.920302    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b06916f-f4be-4bee-b908-7e7ca985c474-proxy-tls\") pod \"ensemble-graph-6de22-7f888f584b-85sxv\" (UID: \"4b06916f-f4be-4bee-b908-7e7ca985c474\") " pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:37:48.130976 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:48.130950    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:37:48.260578 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:48.260553    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"]
Jul 24 21:37:48.262536 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:37:48.262512    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b06916f_f4be_4bee_b908_7e7ca985c474.slice/crio-79a8c7fd4f364957d630dbdf75d3ea7036b482d5b76107f7c85d4f89e950e879 WatchSource:0}: Error finding container 79a8c7fd4f364957d630dbdf75d3ea7036b482d5b76107f7c85d4f89e950e879: Status 404 returned error can't find the container with id 79a8c7fd4f364957d630dbdf75d3ea7036b482d5b76107f7c85d4f89e950e879
Jul 24 21:37:48.832733 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:48.832700    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" podUID="e149d103-691c-4e5c-ac97-f5051ec0dea5" containerName="sequence-graph-f761f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:37:48.832908 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:48.832784    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:37:48.927155 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:48.927117    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" event={"ID":"4b06916f-f4be-4bee-b908-7e7ca985c474","Type":"ContainerStarted","Data":"5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873"}
Jul 24 21:37:48.927321 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:48.927167    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" event={"ID":"4b06916f-f4be-4bee-b908-7e7ca985c474","Type":"ContainerStarted","Data":"79a8c7fd4f364957d630dbdf75d3ea7036b482d5b76107f7c85d4f89e950e879"}
Jul 24 21:37:48.927321 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:48.927188    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:37:48.943334 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:48.943288    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" podStartSLOduration=1.943273079 podStartE2EDuration="1.943273079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:37:48.94192835 +0000 UTC m=+1390.973323358" watchObservedRunningTime="2026-07-24 21:37:48.943273079 +0000 UTC m=+1390.974668086"
Jul 24 21:37:53.833599 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:53.833553    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" podUID="e149d103-691c-4e5c-ac97-f5051ec0dea5" containerName="sequence-graph-f761f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:37:54.889903 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:54.889865    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 24 21:37:54.938130 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:54.938107    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:37:58.832982 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:37:58.832948    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" podUID="e149d103-691c-4e5c-ac97-f5051ec0dea5" containerName="sequence-graph-f761f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:38:03.832944 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:03.832902    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" podUID="e149d103-691c-4e5c-ac97-f5051ec0dea5" containerName="sequence-graph-f761f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:38:04.889847 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:04.889809    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 24 21:38:07.013248 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:07.013211    2522 generic.go:356] "Generic (PLEG): container finished" podID="e149d103-691c-4e5c-ac97-f5051ec0dea5" containerID="5aba7874920c7f63dbb1970ff406fb16e1e8763ca869cf45dc54ccb8809e5f53" exitCode=0
Jul 24 21:38:07.013580 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:07.013266    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" event={"ID":"e149d103-691c-4e5c-ac97-f5051ec0dea5","Type":"ContainerDied","Data":"5aba7874920c7f63dbb1970ff406fb16e1e8763ca869cf45dc54ccb8809e5f53"}
Jul 24 21:38:07.033035 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:07.033009    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:38:07.157034 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:07.156963    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e149d103-691c-4e5c-ac97-f5051ec0dea5-openshift-service-ca-bundle\") pod \"e149d103-691c-4e5c-ac97-f5051ec0dea5\" (UID: \"e149d103-691c-4e5c-ac97-f5051ec0dea5\") "
Jul 24 21:38:07.157150 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:07.157056    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e149d103-691c-4e5c-ac97-f5051ec0dea5-proxy-tls\") pod \"e149d103-691c-4e5c-ac97-f5051ec0dea5\" (UID: \"e149d103-691c-4e5c-ac97-f5051ec0dea5\") "
Jul 24 21:38:07.157306 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:07.157279    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e149d103-691c-4e5c-ac97-f5051ec0dea5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e149d103-691c-4e5c-ac97-f5051ec0dea5" (UID: "e149d103-691c-4e5c-ac97-f5051ec0dea5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:38:07.159144 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:07.159115    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e149d103-691c-4e5c-ac97-f5051ec0dea5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e149d103-691c-4e5c-ac97-f5051ec0dea5" (UID: "e149d103-691c-4e5c-ac97-f5051ec0dea5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:38:07.257905 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:07.257884    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e149d103-691c-4e5c-ac97-f5051ec0dea5-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:38:07.257905 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:07.257905    2522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e149d103-691c-4e5c-ac97-f5051ec0dea5-openshift-service-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:38:08.018597 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:08.018560    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"
Jul 24 21:38:08.019044 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:08.018595    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt" event={"ID":"e149d103-691c-4e5c-ac97-f5051ec0dea5","Type":"ContainerDied","Data":"731086bf0425992995d37d6e5302dc924b6c2f1d7321f5139091039070db429c"}
Jul 24 21:38:08.019044 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:08.018657    2522 scope.go:117] "RemoveContainer" containerID="5aba7874920c7f63dbb1970ff406fb16e1e8763ca869cf45dc54ccb8809e5f53"
Jul 24 21:38:08.039529 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:08.039503    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"]
Jul 24 21:38:08.043397 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:08.043372    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f761f-8d6759985-j4lvt"]
Jul 24 21:38:08.407421 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:08.407348    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e149d103-691c-4e5c-ac97-f5051ec0dea5" path="/var/lib/kubelet/pods/e149d103-691c-4e5c-ac97-f5051ec0dea5/volumes"
Jul 24 21:38:12.547454 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:12.547419    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:38:14.890185 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:14.890142    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 24 21:38:19.438948 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:19.438904    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:38:24.891262 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:24.891225    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:38:37.212218 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.212145    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:38:37.245201 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.245168    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"]
Jul 24 21:38:37.245503 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.245491    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e149d103-691c-4e5c-ac97-f5051ec0dea5" containerName="sequence-graph-f761f"
Jul 24 21:38:37.245553 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.245505    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="e149d103-691c-4e5c-ac97-f5051ec0dea5" containerName="sequence-graph-f761f"
Jul 24 21:38:37.245598 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.245579    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="e149d103-691c-4e5c-ac97-f5051ec0dea5" containerName="sequence-graph-f761f"
Jul 24 21:38:37.248451 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.248435    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:38:37.253066 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.253045    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-59fd1-kube-rbac-proxy-sar-config\""
Jul 24 21:38:37.253317 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.253302    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-59fd1-serving-cert\""
Jul 24 21:38:37.279063 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.279037    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"]
Jul 24 21:38:37.377592 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.377563    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff8de588-b741-4b01-a4bd-bafee823b3fc-proxy-tls\") pod \"sequence-graph-59fd1-566c4755cc-q4lp4\" (UID: \"ff8de588-b741-4b01-a4bd-bafee823b3fc\") " pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:38:37.377767 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.377643    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff8de588-b741-4b01-a4bd-bafee823b3fc-openshift-service-ca-bundle\") pod \"sequence-graph-59fd1-566c4755cc-q4lp4\" (UID: \"ff8de588-b741-4b01-a4bd-bafee823b3fc\") " pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:38:37.475694 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.475601    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:38:37.478530 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.478509    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff8de588-b741-4b01-a4bd-bafee823b3fc-openshift-service-ca-bundle\") pod \"sequence-graph-59fd1-566c4755cc-q4lp4\" (UID: \"ff8de588-b741-4b01-a4bd-bafee823b3fc\") " pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:38:37.478651 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.478566    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff8de588-b741-4b01-a4bd-bafee823b3fc-proxy-tls\") pod \"sequence-graph-59fd1-566c4755cc-q4lp4\" (UID: \"ff8de588-b741-4b01-a4bd-bafee823b3fc\") " pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:38:37.479115 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.479095    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff8de588-b741-4b01-a4bd-bafee823b3fc-openshift-service-ca-bundle\") pod \"sequence-graph-59fd1-566c4755cc-q4lp4\" (UID: \"ff8de588-b741-4b01-a4bd-bafee823b3fc\") " pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:38:37.481177 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.481157    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff8de588-b741-4b01-a4bd-bafee823b3fc-proxy-tls\") pod \"sequence-graph-59fd1-566c4755cc-q4lp4\" (UID: \"ff8de588-b741-4b01-a4bd-bafee823b3fc\") " pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:38:37.558719 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.558702    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:38:37.707492 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:37.707469    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"]
Jul 24 21:38:37.709699 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:38:37.709653    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8de588_b741_4b01_a4bd_bafee823b3fc.slice/crio-3cce8c7d0a839dc489a8eca3da7b7b73508604ca4c9c70698509a2d0548b2ece WatchSource:0}: Error finding container 3cce8c7d0a839dc489a8eca3da7b7b73508604ca4c9c70698509a2d0548b2ece: Status 404 returned error can't find the container with id 3cce8c7d0a839dc489a8eca3da7b7b73508604ca4c9c70698509a2d0548b2ece
Jul 24 21:38:38.160085 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:38.160008    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" event={"ID":"ff8de588-b741-4b01-a4bd-bafee823b3fc","Type":"ContainerStarted","Data":"1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76"}
Jul 24 21:38:38.160085 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:38.160058    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" event={"ID":"ff8de588-b741-4b01-a4bd-bafee823b3fc","Type":"ContainerStarted","Data":"3cce8c7d0a839dc489a8eca3da7b7b73508604ca4c9c70698509a2d0548b2ece"}
Jul 24 21:38:38.160274 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:38.160119    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:38:38.195829 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:38.195786    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" podStartSLOduration=1.195772877 podStartE2EDuration="1.195772877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:38:38.194200342 +0000 UTC m=+1440.225595351" watchObservedRunningTime="2026-07-24 21:38:38.195772877 +0000 UTC m=+1440.227167886"
Jul 24 21:38:44.171235 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:38:44.171207    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:40:20.555880 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:40:20.555842    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:40:20.640864 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:40:20.640828    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:46:01.952159 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:01.952077    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"]
Jul 24 21:46:01.952754 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:01.952319    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" podUID="4b06916f-f4be-4bee-b908-7e7ca985c474" containerName="ensemble-graph-6de22" containerID="cri-o://5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873" gracePeriod=30
Jul 24 21:46:02.013941 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.013907    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:46:02.288160 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.288086    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"]
Jul 24 21:46:02.291617 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.291592    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:02.294141 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.294118    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9739a-predictor-serving-cert\""
Jul 24 21:46:02.294663 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.294641    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9739a-kube-rbac-proxy-sar-config\""
Jul 24 21:46:02.301646 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.301619    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"]
Jul 24 21:46:02.348215 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.348180    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"]
Jul 24 21:46:02.348449 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.348424    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kserve-container" containerID="cri-o://61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376" gracePeriod=30
Jul 24 21:46:02.348566 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.348511    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kube-rbac-proxy" containerID="cri-o://afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6" gracePeriod=30
Jul 24 21:46:02.428460 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.428431    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2lp5\" (UniqueName: \"kubernetes.io/projected/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-api-access-v2lp5\") pod \"error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") " pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:02.428566 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.428496    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") " pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:02.428566 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.428523    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-proxy-tls\") pod \"error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") " pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:02.528913 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.528887    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") " pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:02.529026 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.528932    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-proxy-tls\") pod \"error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") " pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:02.529108 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.529082    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lp5\" (UniqueName: \"kubernetes.io/projected/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-api-access-v2lp5\") pod \"error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") " pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:02.529564 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.529544    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") " pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:02.531452 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.531436    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-proxy-tls\") pod \"error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") " pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:02.537466 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.537445    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2lp5\" (UniqueName: \"kubernetes.io/projected/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-api-access-v2lp5\") pod \"error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") " pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:02.601941 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.601886    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:02.735143 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:02.735116    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"]
Jul 24 21:46:02.737577 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:46:02.737549    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d56fde1_16b4_4576_9de3_2ac4af98b1aa.slice/crio-4c0fdb76b9e356fe7c38b40002b995d5845c74f38d319044c098cfe5def8a868 WatchSource:0}: Error finding container 4c0fdb76b9e356fe7c38b40002b995d5845c74f38d319044c098cfe5def8a868: Status 404 returned error can't find the container with id 4c0fdb76b9e356fe7c38b40002b995d5845c74f38d319044c098cfe5def8a868
Jul 24 21:46:03.236061 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:03.236022    2522 generic.go:356] "Generic (PLEG): container finished" podID="8782f4dd-7890-4128-aa45-087f880965a6" containerID="afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6" exitCode=2
Jul 24 21:46:03.236436 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:03.236087    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" event={"ID":"8782f4dd-7890-4128-aa45-087f880965a6","Type":"ContainerDied","Data":"afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6"}
Jul 24 21:46:03.237823 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:03.237797    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" event={"ID":"1d56fde1-16b4-4576-9de3-2ac4af98b1aa","Type":"ContainerStarted","Data":"ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677"}
Jul 24 21:46:03.237944 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:03.237854    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" event={"ID":"1d56fde1-16b4-4576-9de3-2ac4af98b1aa","Type":"ContainerStarted","Data":"5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f"}
Jul 24 21:46:03.237944 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:03.237883    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" event={"ID":"1d56fde1-16b4-4576-9de3-2ac4af98b1aa","Type":"ContainerStarted","Data":"4c0fdb76b9e356fe7c38b40002b995d5845c74f38d319044c098cfe5def8a868"}
Jul 24 21:46:03.238080 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:03.237960    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:03.269487 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:03.269439    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" podStartSLOduration=1.269428135 podStartE2EDuration="1.269428135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:46:03.268137981 +0000 UTC m=+1885.299532989" watchObservedRunningTime="2026-07-24 21:46:03.269428135 +0000 UTC m=+1885.300823143"
Jul 24 21:46:03.290480 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:03.290445    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:46:04.244801 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:04.244771    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:04.246194 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:04.246156    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 24 21:46:04.645098 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:04.645006    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Jul 24 21:46:04.649482 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:04.649463    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 24 21:46:04.936043 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:04.936004    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" podUID="4b06916f-f4be-4bee-b908-7e7ca985c474" containerName="ensemble-graph-6de22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:46:05.251057 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:05.250965    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 24 21:46:05.509099 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:05.509045    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:46:05.654198 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:05.654165    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8782f4dd-7890-4128-aa45-087f880965a6-kube-rbac-proxy-sar-config\") pod \"8782f4dd-7890-4128-aa45-087f880965a6\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") "
Jul 24 21:46:05.654408 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:05.654234    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8782f4dd-7890-4128-aa45-087f880965a6-proxy-tls\") pod \"8782f4dd-7890-4128-aa45-087f880965a6\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") "
Jul 24 21:46:05.654408 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:05.654288    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2gp7\" (UniqueName: \"kubernetes.io/projected/8782f4dd-7890-4128-aa45-087f880965a6-kube-api-access-q2gp7\") pod \"8782f4dd-7890-4128-aa45-087f880965a6\" (UID: \"8782f4dd-7890-4128-aa45-087f880965a6\") "
Jul 24 21:46:05.654616 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:05.654512    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8782f4dd-7890-4128-aa45-087f880965a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8782f4dd-7890-4128-aa45-087f880965a6" (UID: "8782f4dd-7890-4128-aa45-087f880965a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:46:05.656552 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:05.656527    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8782f4dd-7890-4128-aa45-087f880965a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8782f4dd-7890-4128-aa45-087f880965a6" (UID: "8782f4dd-7890-4128-aa45-087f880965a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:46:05.656720 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:05.656577    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8782f4dd-7890-4128-aa45-087f880965a6-kube-api-access-q2gp7" (OuterVolumeSpecName: "kube-api-access-q2gp7") pod "8782f4dd-7890-4128-aa45-087f880965a6" (UID: "8782f4dd-7890-4128-aa45-087f880965a6"). InnerVolumeSpecName "kube-api-access-q2gp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:46:05.755428 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:05.755392    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2gp7\" (UniqueName: \"kubernetes.io/projected/8782f4dd-7890-4128-aa45-087f880965a6-kube-api-access-q2gp7\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:46:05.755428 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:05.755420    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8782f4dd-7890-4128-aa45-087f880965a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:46:05.755428 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:05.755430    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8782f4dd-7890-4128-aa45-087f880965a6-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:46:06.256427 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.256391    2522 generic.go:356] "Generic (PLEG): container finished" podID="8782f4dd-7890-4128-aa45-087f880965a6" containerID="61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376" exitCode=0
Jul 24 21:46:06.256844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.256448    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" event={"ID":"8782f4dd-7890-4128-aa45-087f880965a6","Type":"ContainerDied","Data":"61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376"}
Jul 24 21:46:06.256844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.256462    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"
Jul 24 21:46:06.256844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.256539    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245" event={"ID":"8782f4dd-7890-4128-aa45-087f880965a6","Type":"ContainerDied","Data":"35812cf7c83377dc99ec3fb019d189eb72e7bc210fea499a658bd053c05d8e0f"}
Jul 24 21:46:06.256844 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.256579    2522 scope.go:117] "RemoveContainer" containerID="afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6"
Jul 24 21:46:06.265826 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.265810    2522 scope.go:117] "RemoveContainer" containerID="61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376"
Jul 24 21:46:06.273187 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.273167    2522 scope.go:117] "RemoveContainer" containerID="afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6"
Jul 24 21:46:06.273423 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:46:06.273407    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6\": container with ID starting with afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6 not found: ID does not exist" containerID="afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6"
Jul 24 21:46:06.273468 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.273431    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6"} err="failed to get container status \"afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6\": rpc error: code = NotFound desc = could not find container \"afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6\": container with ID starting with afd8cbf251de52bec39654d81af8c6f021db2a508deba39a6af3444c72d8adf6 not found: ID does not exist"
Jul 24 21:46:06.273468 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.273445    2522 scope.go:117] "RemoveContainer" containerID="61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376"
Jul 24 21:46:06.273638 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:46:06.273623    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376\": container with ID starting with 61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376 not found: ID does not exist" containerID="61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376"
Jul 24 21:46:06.273737 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.273641    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376"} err="failed to get container status \"61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376\": rpc error: code = NotFound desc = could not find container \"61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376\": container with ID starting with 61779b3b75506aaa134a23062836d3163265f9046efd7b71fac43b71fe24e376 not found: ID does not exist"
Jul 24 21:46:06.277358 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.277336    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"]
Jul 24 21:46:06.279299 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.279280    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6de22-predictor-6c4f986f4f-dx245"]
Jul 24 21:46:06.408250 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:06.408224    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8782f4dd-7890-4128-aa45-087f880965a6" path="/var/lib/kubelet/pods/8782f4dd-7890-4128-aa45-087f880965a6/volumes"
Jul 24 21:46:09.935918 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:09.935888    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" podUID="4b06916f-f4be-4bee-b908-7e7ca985c474" containerName="ensemble-graph-6de22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:46:10.256632 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:10.256547    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:10.257033 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:10.257004    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 24 21:46:14.936057 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:14.936016    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" podUID="4b06916f-f4be-4bee-b908-7e7ca985c474" containerName="ensemble-graph-6de22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:46:14.936446 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:14.936111    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:46:19.935692 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:19.935643    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" podUID="4b06916f-f4be-4bee-b908-7e7ca985c474" containerName="ensemble-graph-6de22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:46:20.257663 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:20.257583    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 24 21:46:24.936393 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:24.936357    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" podUID="4b06916f-f4be-4bee-b908-7e7ca985c474" containerName="ensemble-graph-6de22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:46:29.935389 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:29.935351    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" podUID="4b06916f-f4be-4bee-b908-7e7ca985c474" containerName="ensemble-graph-6de22" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:46:30.257626 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:30.257590    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 24 21:46:32.097913 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.097884    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:46:32.247794 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.247718    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b06916f-f4be-4bee-b908-7e7ca985c474-proxy-tls\") pod \"4b06916f-f4be-4bee-b908-7e7ca985c474\" (UID: \"4b06916f-f4be-4bee-b908-7e7ca985c474\") "
Jul 24 21:46:32.247936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.247850    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b06916f-f4be-4bee-b908-7e7ca985c474-openshift-service-ca-bundle\") pod \"4b06916f-f4be-4bee-b908-7e7ca985c474\" (UID: \"4b06916f-f4be-4bee-b908-7e7ca985c474\") "
Jul 24 21:46:32.248208 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.248179    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b06916f-f4be-4bee-b908-7e7ca985c474-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4b06916f-f4be-4bee-b908-7e7ca985c474" (UID: "4b06916f-f4be-4bee-b908-7e7ca985c474"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:46:32.249756 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.249728    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b06916f-f4be-4bee-b908-7e7ca985c474-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b06916f-f4be-4bee-b908-7e7ca985c474" (UID: "4b06916f-f4be-4bee-b908-7e7ca985c474"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:46:32.348886 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.348863    2522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b06916f-f4be-4bee-b908-7e7ca985c474-openshift-service-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:46:32.348886 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.348885    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b06916f-f4be-4bee-b908-7e7ca985c474-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:46:32.374061 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.374027    2522 generic.go:356] "Generic (PLEG): container finished" podID="4b06916f-f4be-4bee-b908-7e7ca985c474" containerID="5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873" exitCode=0
Jul 24 21:46:32.374159 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.374069    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"
Jul 24 21:46:32.374159 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.374077    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" event={"ID":"4b06916f-f4be-4bee-b908-7e7ca985c474","Type":"ContainerDied","Data":"5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873"}
Jul 24 21:46:32.374159 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.374137    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv" event={"ID":"4b06916f-f4be-4bee-b908-7e7ca985c474","Type":"ContainerDied","Data":"79a8c7fd4f364957d630dbdf75d3ea7036b482d5b76107f7c85d4f89e950e879"}
Jul 24 21:46:32.374290 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.374170    2522 scope.go:117] "RemoveContainer" containerID="5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873"
Jul 24 21:46:32.382795 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.382779    2522 scope.go:117] "RemoveContainer" containerID="5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873"
Jul 24 21:46:32.383061 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:46:32.383043    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873\": container with ID starting with 5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873 not found: ID does not exist" containerID="5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873"
Jul 24 21:46:32.383130 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.383069    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873"} err="failed to get container status \"5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873\": rpc error: code = NotFound desc = could not find container \"5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873\": container with ID starting with 5bac6f6a512aaa1b1b18c7211c28a342a9f897e6e155b06aee4f014a66de1873 not found: ID does not exist"
Jul 24 21:46:32.395116 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.395097    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"]
Jul 24 21:46:32.400817 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.400797    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6de22-7f888f584b-85sxv"]
Jul 24 21:46:32.406490 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:32.406464    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b06916f-f4be-4bee-b908-7e7ca985c474" path="/var/lib/kubelet/pods/4b06916f-f4be-4bee-b908-7e7ca985c474/volumes"
Jul 24 21:46:40.257590 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:40.257549    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 24 21:46:50.258524 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:50.258491    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:46:51.874052 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:51.874020    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"]
Jul 24 21:46:51.874536 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:51.874210    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" podUID="ff8de588-b741-4b01-a4bd-bafee823b3fc" containerName="sequence-graph-59fd1" containerID="cri-o://1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76" gracePeriod=30
Jul 24 21:46:51.898354 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:51.898324    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:46:52.011165 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.011128    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:46:52.102249 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.102218    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"]
Jul 24 21:46:52.102513 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.102482    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kserve-container" containerID="cri-o://348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca" gracePeriod=30
Jul 24 21:46:52.102662 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.102526    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kube-rbac-proxy" containerID="cri-o://58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db" gracePeriod=30
Jul 24 21:46:52.107047 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.107023    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:46:52.135620 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.135567    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"]
Jul 24 21:46:52.135968 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.135955    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kserve-container"
Jul 24 21:46:52.136016 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.135970    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kserve-container"
Jul 24 21:46:52.136016 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.136000    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b06916f-f4be-4bee-b908-7e7ca985c474" containerName="ensemble-graph-6de22"
Jul 24 21:46:52.136016 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.136007    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b06916f-f4be-4bee-b908-7e7ca985c474" containerName="ensemble-graph-6de22"
Jul 24 21:46:52.136016 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.136013    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kube-rbac-proxy"
Jul 24 21:46:52.136140 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.136018    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kube-rbac-proxy"
Jul 24 21:46:52.136140 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.136076    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kserve-container"
Jul 24 21:46:52.136140 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.136085    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="8782f4dd-7890-4128-aa45-087f880965a6" containerName="kube-rbac-proxy"
Jul 24 21:46:52.136140 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.136093    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b06916f-f4be-4bee-b908-7e7ca985c474" containerName="ensemble-graph-6de22"
Jul 24 21:46:52.139270 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.139254    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:52.141435 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.141416    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a916f-predictor-serving-cert\""
Jul 24 21:46:52.141532 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.141477    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a916f-kube-rbac-proxy-sar-config\""
Jul 24 21:46:52.150655 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.150634    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"]
Jul 24 21:46:52.179657 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.179639    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:46:52.296147 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.296114    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-proxy-tls\") pod \"error-404-isvc-a916f-predictor-5548f55f77-vn8q2\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") " pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:52.296292 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.296170    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a916f-predictor-5548f55f77-vn8q2\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") " pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:52.296292 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.296208    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7qr\" (UniqueName: \"kubernetes.io/projected/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-api-access-lz7qr\") pod \"error-404-isvc-a916f-predictor-5548f55f77-vn8q2\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") " pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:52.339119 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.339095    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:46:52.397027 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.396964    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-proxy-tls\") pod \"error-404-isvc-a916f-predictor-5548f55f77-vn8q2\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") " pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:52.397027 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.396997    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a916f-predictor-5548f55f77-vn8q2\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") " pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:52.397183 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.397026    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7qr\" (UniqueName: \"kubernetes.io/projected/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-api-access-lz7qr\") pod \"error-404-isvc-a916f-predictor-5548f55f77-vn8q2\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") " pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:52.397653 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.397599    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a916f-predictor-5548f55f77-vn8q2\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") " pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:52.399385 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.399369    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-proxy-tls\") pod \"error-404-isvc-a916f-predictor-5548f55f77-vn8q2\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") " pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:52.407376 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.407355    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7qr\" (UniqueName: \"kubernetes.io/projected/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-api-access-lz7qr\") pod \"error-404-isvc-a916f-predictor-5548f55f77-vn8q2\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") " pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:52.442746 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.442659    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:46:52.449440 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.449419    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:52.471880 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.471858    2522 generic.go:356] "Generic (PLEG): container finished" podID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerID="58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db" exitCode=2
Jul 24 21:46:52.471993 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.471907    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" event={"ID":"c21056da-69b4-4ad2-bd14-a65e99a848ed","Type":"ContainerDied","Data":"58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db"}
Jul 24 21:46:52.581154 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:52.581130    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"]
Jul 24 21:46:52.583040 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:46:52.583003    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d3f802_7324_4f66_b47e_4a3ba18fe0aa.slice/crio-de1b4e24950d13d7b9806b9174fdc04a16eb3f84d9ba8ad26c8a9bb3f2d90ca9 WatchSource:0}: Error finding container de1b4e24950d13d7b9806b9174fdc04a16eb3f84d9ba8ad26c8a9bb3f2d90ca9: Status 404 returned error can't find the container with id de1b4e24950d13d7b9806b9174fdc04a16eb3f84d9ba8ad26c8a9bb3f2d90ca9
Jul 24 21:46:53.484633 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:53.484591    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" event={"ID":"40d3f802-7324-4f66-b47e-4a3ba18fe0aa","Type":"ContainerStarted","Data":"d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad"}
Jul 24 21:46:53.485121 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:53.484722    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" event={"ID":"40d3f802-7324-4f66-b47e-4a3ba18fe0aa","Type":"ContainerStarted","Data":"a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec"}
Jul 24 21:46:53.485121 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:53.484762    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" event={"ID":"40d3f802-7324-4f66-b47e-4a3ba18fe0aa","Type":"ContainerStarted","Data":"de1b4e24950d13d7b9806b9174fdc04a16eb3f84d9ba8ad26c8a9bb3f2d90ca9"}
Jul 24 21:46:53.485121 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:53.484799    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:53.504921 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:53.504868    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" podStartSLOduration=1.504853413 podStartE2EDuration="1.504853413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:46:53.503283499 +0000 UTC m=+1935.534678507" watchObservedRunningTime="2026-07-24 21:46:53.504853413 +0000 UTC m=+1935.536248422"
Jul 24 21:46:54.168656 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:54.168614    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" podUID="ff8de588-b741-4b01-a4bd-bafee823b3fc" containerName="sequence-graph-59fd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:46:54.490931 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:54.490891    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:46:54.492192 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:54.492167    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 24 21:46:54.951927 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:54.951908    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:46:55.016178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.016148    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bqnh\" (UniqueName: \"kubernetes.io/projected/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-api-access-9bqnh\") pod \"c21056da-69b4-4ad2-bd14-a65e99a848ed\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") "
Jul 24 21:46:55.016178 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.016181    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-rbac-proxy-sar-config\") pod \"c21056da-69b4-4ad2-bd14-a65e99a848ed\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") "
Jul 24 21:46:55.016379 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.016200    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21056da-69b4-4ad2-bd14-a65e99a848ed-proxy-tls\") pod \"c21056da-69b4-4ad2-bd14-a65e99a848ed\" (UID: \"c21056da-69b4-4ad2-bd14-a65e99a848ed\") "
Jul 24 21:46:55.016537 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.016512    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c21056da-69b4-4ad2-bd14-a65e99a848ed" (UID: "c21056da-69b4-4ad2-bd14-a65e99a848ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:46:55.018346 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.018322    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21056da-69b4-4ad2-bd14-a65e99a848ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c21056da-69b4-4ad2-bd14-a65e99a848ed" (UID: "c21056da-69b4-4ad2-bd14-a65e99a848ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:46:55.018445 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.018399    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-api-access-9bqnh" (OuterVolumeSpecName: "kube-api-access-9bqnh") pod "c21056da-69b4-4ad2-bd14-a65e99a848ed" (UID: "c21056da-69b4-4ad2-bd14-a65e99a848ed"). InnerVolumeSpecName "kube-api-access-9bqnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:46:55.116942 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.116871    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bqnh\" (UniqueName: \"kubernetes.io/projected/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-api-access-9bqnh\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:46:55.116942 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.116895    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c21056da-69b4-4ad2-bd14-a65e99a848ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:46:55.116942 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.116904    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c21056da-69b4-4ad2-bd14-a65e99a848ed-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:46:55.496823 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.496793    2522 generic.go:356] "Generic (PLEG): container finished" podID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerID="348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca" exitCode=0
Jul 24 21:46:55.497195 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.496847    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"
Jul 24 21:46:55.497195 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.496886    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" event={"ID":"c21056da-69b4-4ad2-bd14-a65e99a848ed","Type":"ContainerDied","Data":"348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca"}
Jul 24 21:46:55.497195 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.496950    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" event={"ID":"c21056da-69b4-4ad2-bd14-a65e99a848ed","Type":"ContainerDied","Data":"36eec655de851c36d451fa292a015fc17b10442650f7f145adaa2f1555ba1638"}
Jul 24 21:46:55.497195 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.496978    2522 scope.go:117] "RemoveContainer" containerID="58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db"
Jul 24 21:46:55.497554 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.497527    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 24 21:46:55.507159 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.507145    2522 scope.go:117] "RemoveContainer" containerID="348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca"
Jul 24 21:46:55.515865 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.515844    2522 scope.go:117] "RemoveContainer" containerID="58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db"
Jul 24 21:46:55.516140 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:46:55.516116    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db\": container with ID starting with 58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db not found: ID does not exist" containerID="58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db"
Jul 24 21:46:55.516229 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.516148    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db"} err="failed to get container status \"58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db\": rpc error: code = NotFound desc = could not find container \"58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db\": container with ID starting with 58c26fdc0f7a5708bf36d48c6cfece78776f90bfc227285b586bf49da9a230db not found: ID does not exist"
Jul 24 21:46:55.516229 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.516165    2522 scope.go:117] "RemoveContainer" containerID="348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca"
Jul 24 21:46:55.516427 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:46:55.516407    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca\": container with ID starting with 348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca not found: ID does not exist" containerID="348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca"
Jul 24 21:46:55.516468 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.516435    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca"} err="failed to get container status \"348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca\": rpc error: code = NotFound desc = could not find container \"348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca\": container with ID starting with 348b1d6bda924ebb19be237c12356c379b8a55de1ae12b205d0a6cc37d3edcca not found: ID does not exist"
Jul 24 21:46:55.525434 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.525414    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"]
Jul 24 21:46:55.531595 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.531574    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz"]
Jul 24 21:46:55.885481 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.885403    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 24 21:46:55.890573 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:55.890547    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59fd1-predictor-6c56cf6df7-pknkz" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: i/o timeout"
Jul 24 21:46:56.408611 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:56.408575    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" path="/var/lib/kubelet/pods/c21056da-69b4-4ad2-bd14-a65e99a848ed/volumes"
Jul 24 21:46:56.835710 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:56.835665    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:46:59.168972 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:46:59.168924    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" podUID="ff8de588-b741-4b01-a4bd-bafee823b3fc" containerName="sequence-graph-59fd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:47:00.502056 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:00.502030    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:47:00.502543 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:00.502513    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 24 21:47:02.191375 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.191345    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"]
Jul 24 21:47:02.191786 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.191706    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kserve-container"
Jul 24 21:47:02.191786 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.191721    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kserve-container"
Jul 24 21:47:02.191786 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.191735    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kube-rbac-proxy"
Jul 24 21:47:02.191786 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.191740    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kube-rbac-proxy"
Jul 24 21:47:02.191928 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.191806    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kserve-container"
Jul 24 21:47:02.191928 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.191817    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="c21056da-69b4-4ad2-bd14-a65e99a848ed" containerName="kube-rbac-proxy"
Jul 24 21:47:02.196363 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.196334    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:02.197414 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.197390    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:02.198647 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.198632    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-9739a-serving-cert\""
Jul 24 21:47:02.198874 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.198857    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-9739a-kube-rbac-proxy-sar-config\""
Jul 24 21:47:02.203122 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.203097    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"]
Jul 24 21:47:02.264752 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.264723    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1897e613-6bd4-4107-87ca-669289cd15a9-openshift-service-ca-bundle\") pod \"splitter-graph-9739a-647798446f-ffsxh\" (UID: \"1897e613-6bd4-4107-87ca-669289cd15a9\") " pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:02.264897 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.264787    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls\") pod \"splitter-graph-9739a-647798446f-ffsxh\" (UID: \"1897e613-6bd4-4107-87ca-669289cd15a9\") " pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:02.365276 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.365240    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1897e613-6bd4-4107-87ca-669289cd15a9-openshift-service-ca-bundle\") pod \"splitter-graph-9739a-647798446f-ffsxh\" (UID: \"1897e613-6bd4-4107-87ca-669289cd15a9\") " pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:02.365436 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.365295    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls\") pod \"splitter-graph-9739a-647798446f-ffsxh\" (UID: \"1897e613-6bd4-4107-87ca-669289cd15a9\") " pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:02.365436 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:02.365396    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-9739a-serving-cert: secret "splitter-graph-9739a-serving-cert" not found
Jul 24 21:47:02.365526 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:02.365461    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls podName:1897e613-6bd4-4107-87ca-669289cd15a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:47:02.865444397 +0000 UTC m=+1944.896839383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls") pod "splitter-graph-9739a-647798446f-ffsxh" (UID: "1897e613-6bd4-4107-87ca-669289cd15a9") : secret "splitter-graph-9739a-serving-cert" not found
Jul 24 21:47:02.365874 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.365855    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1897e613-6bd4-4107-87ca-669289cd15a9-openshift-service-ca-bundle\") pod \"splitter-graph-9739a-647798446f-ffsxh\" (UID: \"1897e613-6bd4-4107-87ca-669289cd15a9\") " pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:02.869568 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:02.869534    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls\") pod \"splitter-graph-9739a-647798446f-ffsxh\" (UID: \"1897e613-6bd4-4107-87ca-669289cd15a9\") " pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:02.869753 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:02.869697    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-9739a-serving-cert: secret "splitter-graph-9739a-serving-cert" not found
Jul 24 21:47:02.869807 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:02.869759    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls podName:1897e613-6bd4-4107-87ca-669289cd15a9 nodeName:}" failed. No retries permitted until 2026-07-24 21:47:03.869744351 +0000 UTC m=+1945.901139338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls") pod "splitter-graph-9739a-647798446f-ffsxh" (UID: "1897e613-6bd4-4107-87ca-669289cd15a9") : secret "splitter-graph-9739a-serving-cert" not found
Jul 24 21:47:03.033819 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:03.033769    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:03.877204 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:03.877163    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls\") pod \"splitter-graph-9739a-647798446f-ffsxh\" (UID: \"1897e613-6bd4-4107-87ca-669289cd15a9\") " pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:03.879745 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:03.879726    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls\") pod \"splitter-graph-9739a-647798446f-ffsxh\" (UID: \"1897e613-6bd4-4107-87ca-669289cd15a9\") " pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:04.008654 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:04.008623    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:04.141015 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:04.140992    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"]
Jul 24 21:47:04.144108 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:47:04.144079    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1897e613_6bd4_4107_87ca_669289cd15a9.slice/crio-50b5cf113dd52edd339999a69a5374cc923dd73765a8e4af832347c5f20e3205 WatchSource:0}: Error finding container 50b5cf113dd52edd339999a69a5374cc923dd73765a8e4af832347c5f20e3205: Status 404 returned error can't find the container with id 50b5cf113dd52edd339999a69a5374cc923dd73765a8e4af832347c5f20e3205
Jul 24 21:47:04.169894 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:04.169863    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" podUID="ff8de588-b741-4b01-a4bd-bafee823b3fc" containerName="sequence-graph-59fd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:47:04.170019 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:04.169967    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:47:04.541454 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:04.541421    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" event={"ID":"1897e613-6bd4-4107-87ca-669289cd15a9","Type":"ContainerStarted","Data":"165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118"}
Jul 24 21:47:04.541720 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:04.541476    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" event={"ID":"1897e613-6bd4-4107-87ca-669289cd15a9","Type":"ContainerStarted","Data":"50b5cf113dd52edd339999a69a5374cc923dd73765a8e4af832347c5f20e3205"}
Jul 24 21:47:04.558856 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:04.558813    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" podStartSLOduration=2.5587986000000003 podStartE2EDuration="2.5587986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:47:04.55796922 +0000 UTC m=+1946.589364229" watchObservedRunningTime="2026-07-24 21:47:04.5587986 +0000 UTC m=+1946.590193755"
Jul 24 21:47:05.545832 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:05.545805    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:09.168658 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:09.168622    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" podUID="ff8de588-b741-4b01-a4bd-bafee823b3fc" containerName="sequence-graph-59fd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:47:10.503168 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:10.503129    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 24 21:47:11.555091 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:11.555060    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:12.282838 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.282800    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"]
Jul 24 21:47:12.283027 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.282976    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" podUID="1897e613-6bd4-4107-87ca-669289cd15a9" containerName="splitter-graph-9739a" containerID="cri-o://165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118" gracePeriod=30
Jul 24 21:47:12.303286 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.303261    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:12.365827 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.365801    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"]
Jul 24 21:47:12.366047 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.366023    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kserve-container" containerID="cri-o://5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f" gracePeriod=30
Jul 24 21:47:12.366145 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.366043    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kube-rbac-proxy" containerID="cri-o://ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677" gracePeriod=30
Jul 24 21:47:12.389406 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.389376    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:12.456766 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.456724    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:12.554262 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.554168    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:12.558542 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.558519    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"]
Jul 24 21:47:12.562136 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.562118    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:12.564574 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.564553    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bce0e-predictor-serving-cert\""
Jul 24 21:47:12.564574 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.564567    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bce0e-kube-rbac-proxy-sar-config\""
Jul 24 21:47:12.574493 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.574470    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"]
Jul 24 21:47:12.579030 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.579007    2522 generic.go:356] "Generic (PLEG): container finished" podID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerID="ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677" exitCode=2
Jul 24 21:47:12.579124 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.579064    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" event={"ID":"1d56fde1-16b4-4576-9de3-2ac4af98b1aa","Type":"ContainerDied","Data":"ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677"}
Jul 24 21:47:12.610583 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.610557    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:12.647350 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.647330    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e26efb-19f4-4911-95e3-91a0705615cd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") " pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:12.647475 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.647355    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxfqd\" (UniqueName: \"kubernetes.io/projected/72e26efb-19f4-4911-95e3-91a0705615cd-kube-api-access-vxfqd\") pod \"error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") " pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:12.647475 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.647397    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e26efb-19f4-4911-95e3-91a0705615cd-proxy-tls\") pod \"error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") " pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:12.687770 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.687734    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:12.748105 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.748079    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e26efb-19f4-4911-95e3-91a0705615cd-proxy-tls\") pod \"error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") " pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:12.748217 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.748132    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e26efb-19f4-4911-95e3-91a0705615cd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") " pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:12.748273 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.748250    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxfqd\" (UniqueName: \"kubernetes.io/projected/72e26efb-19f4-4911-95e3-91a0705615cd-kube-api-access-vxfqd\") pod \"error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") " pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:12.748713 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.748695    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e26efb-19f4-4911-95e3-91a0705615cd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") " pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:12.750700 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.750662    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e26efb-19f4-4911-95e3-91a0705615cd-proxy-tls\") pod \"error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") " pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:12.757021 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.756999    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxfqd\" (UniqueName: \"kubernetes.io/projected/72e26efb-19f4-4911-95e3-91a0705615cd-kube-api-access-vxfqd\") pod \"error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") " pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:12.872851 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:12.872792    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:13.004227 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:13.004198    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"]
Jul 24 21:47:13.006262 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:47:13.006237    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e26efb_19f4_4911_95e3_91a0705615cd.slice/crio-514685e9aaec6161d19997e37aaa0232ae419a0ba79496bb9afcdfa6fcb5230c WatchSource:0}: Error finding container 514685e9aaec6161d19997e37aaa0232ae419a0ba79496bb9afcdfa6fcb5230c: Status 404 returned error can't find the container with id 514685e9aaec6161d19997e37aaa0232ae419a0ba79496bb9afcdfa6fcb5230c
Jul 24 21:47:13.587765 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:13.587719    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" event={"ID":"72e26efb-19f4-4911-95e3-91a0705615cd","Type":"ContainerStarted","Data":"b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887"}
Jul 24 21:47:13.588212 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:13.587793    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" event={"ID":"72e26efb-19f4-4911-95e3-91a0705615cd","Type":"ContainerStarted","Data":"27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77"}
Jul 24 21:47:13.588212 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:13.587826    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" event={"ID":"72e26efb-19f4-4911-95e3-91a0705615cd","Type":"ContainerStarted","Data":"514685e9aaec6161d19997e37aaa0232ae419a0ba79496bb9afcdfa6fcb5230c"}
Jul 24 21:47:13.588212 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:13.587856    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:13.588212 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:13.587884    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:13.589647 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:13.589614    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 24 21:47:13.621982 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:13.621938    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" podStartSLOduration=1.62192444 podStartE2EDuration="1.62192444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:47:13.620822485 +0000 UTC m=+1955.652217492" watchObservedRunningTime="2026-07-24 21:47:13.62192444 +0000 UTC m=+1955.653319448"
Jul 24 21:47:14.168600 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:14.168565    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" podUID="ff8de588-b741-4b01-a4bd-bafee823b3fc" containerName="sequence-graph-59fd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:47:14.593420 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:14.593377    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 24 21:47:15.251710 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.251657    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Jul 24 21:47:15.416868 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.416848    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:47:15.470925 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.470902    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-proxy-tls\") pod \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") "
Jul 24 21:47:15.471043 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.470946    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-rbac-proxy-sar-config\") pod \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") "
Jul 24 21:47:15.471092 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.471042    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2lp5\" (UniqueName: \"kubernetes.io/projected/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-api-access-v2lp5\") pod \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\" (UID: \"1d56fde1-16b4-4576-9de3-2ac4af98b1aa\") "
Jul 24 21:47:15.471357 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.471330    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d56fde1-16b4-4576-9de3-2ac4af98b1aa" (UID: "1d56fde1-16b4-4576-9de3-2ac4af98b1aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:47:15.473148 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.473126    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-api-access-v2lp5" (OuterVolumeSpecName: "kube-api-access-v2lp5") pod "1d56fde1-16b4-4576-9de3-2ac4af98b1aa" (UID: "1d56fde1-16b4-4576-9de3-2ac4af98b1aa"). InnerVolumeSpecName "kube-api-access-v2lp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:47:15.473246 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.473209    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d56fde1-16b4-4576-9de3-2ac4af98b1aa" (UID: "1d56fde1-16b4-4576-9de3-2ac4af98b1aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:47:15.572456 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.572429    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:47:15.572456 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.572453    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2lp5\" (UniqueName: \"kubernetes.io/projected/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-kube-api-access-v2lp5\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:47:15.572597 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.572469    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d56fde1-16b4-4576-9de3-2ac4af98b1aa-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:47:15.598606 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.598580    2522 generic.go:356] "Generic (PLEG): container finished" podID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerID="5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f" exitCode=0
Jul 24 21:47:15.598938 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.598624    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"
Jul 24 21:47:15.598938 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.598656    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" event={"ID":"1d56fde1-16b4-4576-9de3-2ac4af98b1aa","Type":"ContainerDied","Data":"5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f"}
Jul 24 21:47:15.598938 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.598730    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh" event={"ID":"1d56fde1-16b4-4576-9de3-2ac4af98b1aa","Type":"ContainerDied","Data":"4c0fdb76b9e356fe7c38b40002b995d5845c74f38d319044c098cfe5def8a868"}
Jul 24 21:47:15.598938 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.598746    2522 scope.go:117] "RemoveContainer" containerID="ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677"
Jul 24 21:47:15.611243 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.611225    2522 scope.go:117] "RemoveContainer" containerID="5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f"
Jul 24 21:47:15.619758 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.619727    2522 scope.go:117] "RemoveContainer" containerID="ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677"
Jul 24 21:47:15.620050 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:15.620028    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677\": container with ID starting with ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677 not found: ID does not exist" containerID="ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677"
Jul 24 21:47:15.620132 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.620060    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677"} err="failed to get container status \"ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677\": rpc error: code = NotFound desc = could not find container \"ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677\": container with ID starting with ecc3154452eacb7886897b932b4fff6698b140b011539eef7ac94c9a42064677 not found: ID does not exist"
Jul 24 21:47:15.620132 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.620082    2522 scope.go:117] "RemoveContainer" containerID="5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f"
Jul 24 21:47:15.620370 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:15.620351    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f\": container with ID starting with 5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f not found: ID does not exist" containerID="5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f"
Jul 24 21:47:15.620440 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.620376    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f"} err="failed to get container status \"5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f\": rpc error: code = NotFound desc = could not find container \"5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f\": container with ID starting with 5f0d20e78a2a80bbad131b3eb4bff819cae0831fce410eabd45cabd76680d99f not found: ID does not exist"
Jul 24 21:47:15.625043 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.625021    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"]
Jul 24 21:47:15.631761 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:15.631741    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9739a-predictor-68dc85c8c7-jj5rh"]
Jul 24 21:47:16.407461 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:16.407432    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" path="/var/lib/kubelet/pods/1d56fde1-16b4-4576-9de3-2ac4af98b1aa/volumes"
Jul 24 21:47:16.554244 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:16.554218    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" podUID="1897e613-6bd4-4107-87ca-669289cd15a9" containerName="splitter-graph-9739a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:47:19.168781 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:19.168746    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" podUID="ff8de588-b741-4b01-a4bd-bafee823b3fc" containerName="sequence-graph-59fd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:47:19.598052 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:19.598029    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:47:19.598486 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:19.598459    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 24 21:47:20.503449 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:20.503415    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 24 21:47:21.553821 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:21.553786    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" podUID="1897e613-6bd4-4107-87ca-669289cd15a9" containerName="splitter-graph-9739a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:47:22.029111 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.029091    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:47:22.126238 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.126153    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff8de588-b741-4b01-a4bd-bafee823b3fc-proxy-tls\") pod \"ff8de588-b741-4b01-a4bd-bafee823b3fc\" (UID: \"ff8de588-b741-4b01-a4bd-bafee823b3fc\") "
Jul 24 21:47:22.126391 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.126290    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff8de588-b741-4b01-a4bd-bafee823b3fc-openshift-service-ca-bundle\") pod \"ff8de588-b741-4b01-a4bd-bafee823b3fc\" (UID: \"ff8de588-b741-4b01-a4bd-bafee823b3fc\") "
Jul 24 21:47:22.126615 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.126593    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8de588-b741-4b01-a4bd-bafee823b3fc-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ff8de588-b741-4b01-a4bd-bafee823b3fc" (UID: "ff8de588-b741-4b01-a4bd-bafee823b3fc"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:47:22.128286 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.128242    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff8de588-b741-4b01-a4bd-bafee823b3fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff8de588-b741-4b01-a4bd-bafee823b3fc" (UID: "ff8de588-b741-4b01-a4bd-bafee823b3fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:47:22.226991 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.226965    2522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff8de588-b741-4b01-a4bd-bafee823b3fc-openshift-service-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:47:22.226991 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.226986    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff8de588-b741-4b01-a4bd-bafee823b3fc-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:47:22.636629 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.636588    2522 generic.go:356] "Generic (PLEG): container finished" podID="ff8de588-b741-4b01-a4bd-bafee823b3fc" containerID="1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76" exitCode=0
Jul 24 21:47:22.637107 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.636640    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" event={"ID":"ff8de588-b741-4b01-a4bd-bafee823b3fc","Type":"ContainerDied","Data":"1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76"}
Jul 24 21:47:22.637107 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.636646    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"
Jul 24 21:47:22.637107 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.636750    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4" event={"ID":"ff8de588-b741-4b01-a4bd-bafee823b3fc","Type":"ContainerDied","Data":"3cce8c7d0a839dc489a8eca3da7b7b73508604ca4c9c70698509a2d0548b2ece"}
Jul 24 21:47:22.637107 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.636787    2522 scope.go:117] "RemoveContainer" containerID="1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76"
Jul 24 21:47:22.645253 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.645233    2522 scope.go:117] "RemoveContainer" containerID="1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76"
Jul 24 21:47:22.645503 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:22.645484    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76\": container with ID starting with 1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76 not found: ID does not exist" containerID="1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76"
Jul 24 21:47:22.645553 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.645515    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76"} err="failed to get container status \"1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76\": rpc error: code = NotFound desc = could not find container \"1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76\": container with ID starting with 1bc698e33443a1e32cd395d980eff2c6e151cb505becb083a3e70062d2e51c76 not found: ID does not exist"
Jul 24 21:47:22.653463 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.653442    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"]
Jul 24 21:47:22.657590 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:22.657570    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-59fd1-566c4755cc-q4lp4"]
Jul 24 21:47:24.407279 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:24.407246    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff8de588-b741-4b01-a4bd-bafee823b3fc" path="/var/lib/kubelet/pods/ff8de588-b741-4b01-a4bd-bafee823b3fc/volumes"
Jul 24 21:47:26.553357 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:26.553322    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" podUID="1897e613-6bd4-4107-87ca-669289cd15a9" containerName="splitter-graph-9739a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:47:26.553769 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:26.553416    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:29.598639 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:29.598595    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 24 21:47:30.502540 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:30.502503    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 24 21:47:31.553600 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:31.553565    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" podUID="1897e613-6bd4-4107-87ca-669289cd15a9" containerName="splitter-graph-9739a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:47:36.553487 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:36.553437    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" podUID="1897e613-6bd4-4107-87ca-669289cd15a9" containerName="splitter-graph-9739a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:47:39.599371 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:39.599326    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 24 21:47:40.503788 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:40.503762    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 21:47:41.553465 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:41.553426    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" podUID="1897e613-6bd4-4107-87ca-669289cd15a9" containerName="splitter-graph-9739a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:47:42.431365 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.431344    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:42.573908 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.573847    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls\") pod \"1897e613-6bd4-4107-87ca-669289cd15a9\" (UID: \"1897e613-6bd4-4107-87ca-669289cd15a9\") "
Jul 24 21:47:42.573908 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.573895    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1897e613-6bd4-4107-87ca-669289cd15a9-openshift-service-ca-bundle\") pod \"1897e613-6bd4-4107-87ca-669289cd15a9\" (UID: \"1897e613-6bd4-4107-87ca-669289cd15a9\") "
Jul 24 21:47:42.574359 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.574230    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1897e613-6bd4-4107-87ca-669289cd15a9-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1897e613-6bd4-4107-87ca-669289cd15a9" (UID: "1897e613-6bd4-4107-87ca-669289cd15a9"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:47:42.576121 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.576096    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1897e613-6bd4-4107-87ca-669289cd15a9" (UID: "1897e613-6bd4-4107-87ca-669289cd15a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:47:42.674309 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.674286    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1897e613-6bd4-4107-87ca-669289cd15a9-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:47:42.674309 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.674307    2522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1897e613-6bd4-4107-87ca-669289cd15a9-openshift-service-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:47:42.730920 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.730891    2522 generic.go:356] "Generic (PLEG): container finished" podID="1897e613-6bd4-4107-87ca-669289cd15a9" containerID="165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118" exitCode=0
Jul 24 21:47:42.731032 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.730933    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"
Jul 24 21:47:42.731032 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.730940    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" event={"ID":"1897e613-6bd4-4107-87ca-669289cd15a9","Type":"ContainerDied","Data":"165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118"}
Jul 24 21:47:42.731032 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.730982    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh" event={"ID":"1897e613-6bd4-4107-87ca-669289cd15a9","Type":"ContainerDied","Data":"50b5cf113dd52edd339999a69a5374cc923dd73765a8e4af832347c5f20e3205"}
Jul 24 21:47:42.731032 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.731004    2522 scope.go:117] "RemoveContainer" containerID="165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118"
Jul 24 21:47:42.739971 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.739956    2522 scope.go:117] "RemoveContainer" containerID="165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118"
Jul 24 21:47:42.740230 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:42.740212    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118\": container with ID starting with 165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118 not found: ID does not exist" containerID="165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118"
Jul 24 21:47:42.740301 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.740241    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118"} err="failed to get container status \"165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118\": rpc error: code = NotFound desc = could not find container \"165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118\": container with ID starting with 165e199f174ba85cc657ac191c6b063cceb3bea901ccb43fceb60ab373d41118 not found: ID does not exist"
Jul 24 21:47:42.751535 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.751507    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"]
Jul 24 21:47:42.755559 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:42.755540    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9739a-647798446f-ffsxh"]
Jul 24 21:47:44.407204 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:44.407170    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1897e613-6bd4-4107-87ca-669289cd15a9" path="/var/lib/kubelet/pods/1897e613-6bd4-4107-87ca-669289cd15a9/volumes"
Jul 24 21:47:49.599305 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:49.599270    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 24 21:47:52.092501 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092466    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"]
Jul 24 21:47:52.092936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092824    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kube-rbac-proxy"
Jul 24 21:47:52.092936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092835    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kube-rbac-proxy"
Jul 24 21:47:52.092936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092848    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1897e613-6bd4-4107-87ca-669289cd15a9" containerName="splitter-graph-9739a"
Jul 24 21:47:52.092936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092854    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="1897e613-6bd4-4107-87ca-669289cd15a9" containerName="splitter-graph-9739a"
Jul 24 21:47:52.092936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092863    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kserve-container"
Jul 24 21:47:52.092936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092868    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kserve-container"
Jul 24 21:47:52.092936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092879    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8de588-b741-4b01-a4bd-bafee823b3fc" containerName="sequence-graph-59fd1"
Jul 24 21:47:52.092936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092884    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8de588-b741-4b01-a4bd-bafee823b3fc" containerName="sequence-graph-59fd1"
Jul 24 21:47:52.092936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092933    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kserve-container"
Jul 24 21:47:52.092936 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092941    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="1897e613-6bd4-4107-87ca-669289cd15a9" containerName="splitter-graph-9739a"
Jul 24 21:47:52.093278 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092948    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d56fde1-16b4-4576-9de3-2ac4af98b1aa" containerName="kube-rbac-proxy"
Jul 24 21:47:52.093278 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.092957    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8de588-b741-4b01-a4bd-bafee823b3fc" containerName="sequence-graph-59fd1"
Jul 24 21:47:52.097299 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.097279    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:47:52.099904 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.099884    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a916f-serving-cert\""
Jul 24 21:47:52.099904 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.099898    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a916f-kube-rbac-proxy-sar-config\""
Jul 24 21:47:52.103789 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.103760    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"]
Jul 24 21:47:52.107144 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.107104    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:52.243365 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.243333    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd9e0c35-68db-42f7-a4a9-066869594ece-openshift-service-ca-bundle\") pod \"switch-graph-a916f-6d9bcff485-5rs4f\" (UID: \"cd9e0c35-68db-42f7-a4a9-066869594ece\") " pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:47:52.243518 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.243385    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls\") pod \"switch-graph-a916f-6d9bcff485-5rs4f\" (UID: \"cd9e0c35-68db-42f7-a4a9-066869594ece\") " pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:47:52.343851 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.343776    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls\") pod \"switch-graph-a916f-6d9bcff485-5rs4f\" (UID: \"cd9e0c35-68db-42f7-a4a9-066869594ece\") " pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:47:52.343999 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.343852    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd9e0c35-68db-42f7-a4a9-066869594ece-openshift-service-ca-bundle\") pod \"switch-graph-a916f-6d9bcff485-5rs4f\" (UID: \"cd9e0c35-68db-42f7-a4a9-066869594ece\") " pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:47:52.343999 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:52.343944    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-a916f-serving-cert: secret "switch-graph-a916f-serving-cert" not found
Jul 24 21:47:52.344081 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:52.344034    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls podName:cd9e0c35-68db-42f7-a4a9-066869594ece nodeName:}" failed. No retries permitted until 2026-07-24 21:47:52.844011504 +0000 UTC m=+1994.875406492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls") pod "switch-graph-a916f-6d9bcff485-5rs4f" (UID: "cd9e0c35-68db-42f7-a4a9-066869594ece") : secret "switch-graph-a916f-serving-cert" not found
Jul 24 21:47:52.344430 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.344411    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd9e0c35-68db-42f7-a4a9-066869594ece-openshift-service-ca-bundle\") pod \"switch-graph-a916f-6d9bcff485-5rs4f\" (UID: \"cd9e0c35-68db-42f7-a4a9-066869594ece\") " pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:47:52.848207 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.848179    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls\") pod \"switch-graph-a916f-6d9bcff485-5rs4f\" (UID: \"cd9e0c35-68db-42f7-a4a9-066869594ece\") " pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:47:52.848382 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:52.848325    2522 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-a916f-serving-cert: secret "switch-graph-a916f-serving-cert" not found
Jul 24 21:47:52.848427 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:47:52.848419    2522 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls podName:cd9e0c35-68db-42f7-a4a9-066869594ece nodeName:}" failed. No retries permitted until 2026-07-24 21:47:53.848403834 +0000 UTC m=+1995.879798819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls") pod "switch-graph-a916f-6d9bcff485-5rs4f" (UID: "cd9e0c35-68db-42f7-a4a9-066869594ece") : secret "switch-graph-a916f-serving-cert" not found
Jul 24 21:47:52.928586 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:52.928549    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:53.856606 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:53.856576    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls\") pod \"switch-graph-a916f-6d9bcff485-5rs4f\" (UID: \"cd9e0c35-68db-42f7-a4a9-066869594ece\") " pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:47:53.859173 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:53.859154    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls\") pod \"switch-graph-a916f-6d9bcff485-5rs4f\" (UID: \"cd9e0c35-68db-42f7-a4a9-066869594ece\") " pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:47:53.909108 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:53.909083    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:47:54.040271 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:54.040093    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"]
Jul 24 21:47:54.042716 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:47:54.042695    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9e0c35_68db_42f7_a4a9_066869594ece.slice/crio-a1721c42df9dd81e7c158a23464d4cef53251d5fb27a16147bd1aa6f49158513 WatchSource:0}: Error finding container a1721c42df9dd81e7c158a23464d4cef53251d5fb27a16147bd1aa6f49158513: Status 404 returned error can't find the container with id a1721c42df9dd81e7c158a23464d4cef53251d5fb27a16147bd1aa6f49158513
Jul 24 21:47:54.787041 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:54.787005    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" event={"ID":"cd9e0c35-68db-42f7-a4a9-066869594ece","Type":"ContainerStarted","Data":"ede1fd83b1f29268a597cf3102b05dd4e489ed52c164002498b2690ca5c9948c"}
Jul 24 21:47:54.787254 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:54.787054    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" event={"ID":"cd9e0c35-68db-42f7-a4a9-066869594ece","Type":"ContainerStarted","Data":"a1721c42df9dd81e7c158a23464d4cef53251d5fb27a16147bd1aa6f49158513"}
Jul 24 21:47:54.787254 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:54.787085    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:47:54.807427 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:54.807373    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" podStartSLOduration=2.807358951 podStartE2EDuration="2.807358951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:47:54.805518484 +0000 UTC m=+1996.836913494" watchObservedRunningTime="2026-07-24 21:47:54.807358951 +0000 UTC m=+1996.838753959"
Jul 24 21:47:59.599851 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:47:59.599817    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:48:00.805048 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:00.805016    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 21:48:12.513075 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.513045    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"]
Jul 24 21:48:12.516437 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.516417    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:48:12.519031 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.519005    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-bce0e-kube-rbac-proxy-sar-config\""
Jul 24 21:48:12.519150 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.519095    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-bce0e-serving-cert\""
Jul 24 21:48:12.520238 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.520216    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:48:12.527626 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.527605    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"]
Jul 24 21:48:12.694635 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.694601    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-openshift-service-ca-bundle\") pod \"splitter-graph-bce0e-8f65fd6fd-zlngh\" (UID: \"1c9a9c53-5c0f-42d3-a45b-071d67b40be8\") " pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:48:12.694894 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.694650    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-proxy-tls\") pod \"splitter-graph-bce0e-8f65fd6fd-zlngh\" (UID: \"1c9a9c53-5c0f-42d3-a45b-071d67b40be8\") " pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:48:12.739427 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.739390    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:48:12.795532 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.795477    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-proxy-tls\") pod \"splitter-graph-bce0e-8f65fd6fd-zlngh\" (UID: \"1c9a9c53-5c0f-42d3-a45b-071d67b40be8\") " pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:48:12.795630 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.795553    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-openshift-service-ca-bundle\") pod \"splitter-graph-bce0e-8f65fd6fd-zlngh\" (UID: \"1c9a9c53-5c0f-42d3-a45b-071d67b40be8\") " pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:48:12.796230 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.796200    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-openshift-service-ca-bundle\") pod \"splitter-graph-bce0e-8f65fd6fd-zlngh\" (UID: \"1c9a9c53-5c0f-42d3-a45b-071d67b40be8\") " pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:48:12.798000 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.797982    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-proxy-tls\") pod \"splitter-graph-bce0e-8f65fd6fd-zlngh\" (UID: \"1c9a9c53-5c0f-42d3-a45b-071d67b40be8\") " pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:48:12.827889 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.827858    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:48:12.958225 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:12.958201    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"]
Jul 24 21:48:12.960385 ip-10-0-140-162 kubenswrapper[2522]: W0724 21:48:12.960360    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9a9c53_5c0f_42d3_a45b_071d67b40be8.slice/crio-86c4812d04aeceb8f97e0018b1b0a2745795a6c671758719f043215ded5085ec WatchSource:0}: Error finding container 86c4812d04aeceb8f97e0018b1b0a2745795a6c671758719f043215ded5085ec: Status 404 returned error can't find the container with id 86c4812d04aeceb8f97e0018b1b0a2745795a6c671758719f043215ded5085ec
Jul 24 21:48:13.882788 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:13.882744    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" event={"ID":"1c9a9c53-5c0f-42d3-a45b-071d67b40be8","Type":"ContainerStarted","Data":"008254a1056bf60223aae21e5999e7188d8eb56dea6367b617bf9cb6a9ad64cc"}
Jul 24 21:48:13.883281 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:13.882807    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" event={"ID":"1c9a9c53-5c0f-42d3-a45b-071d67b40be8","Type":"ContainerStarted","Data":"86c4812d04aeceb8f97e0018b1b0a2745795a6c671758719f043215ded5085ec"}
Jul 24 21:48:13.883281 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:13.882834    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:48:13.899852 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:13.899799    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" podStartSLOduration=1.899782826 podStartE2EDuration="1.899782826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:48:13.898620061 +0000 UTC m=+2015.930015070" watchObservedRunningTime="2026-07-24 21:48:13.899782826 +0000 UTC m=+2015.931177833"
Jul 24 21:48:19.895285 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:48:19.895251    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:52:47.044398 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:52:47.044358    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:52:48.463897 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:52:48.463864    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:55:32.426787 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:55:32.426745    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:55:34.395744 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:55:34.395694    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:55:46.302042 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:55:46.302003    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:56:27.206954 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:27.206917    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"]
Jul 24 21:56:27.209481 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:27.207513    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" podUID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" containerName="splitter-graph-bce0e" containerID="cri-o://008254a1056bf60223aae21e5999e7188d8eb56dea6367b617bf9cb6a9ad64cc" gracePeriod=30
Jul 24 21:56:27.226493 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:27.226439    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:56:27.352114 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:27.352081    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:56:27.359996 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:27.359964    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"]
Jul 24 21:56:27.360207 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:27.360181    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kserve-container" containerID="cri-o://27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77" gracePeriod=30
Jul 24 21:56:27.360302 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:27.360220    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kube-rbac-proxy" containerID="cri-o://b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887" gracePeriod=30
Jul 24 21:56:27.411805 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:27.411784    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 21:56:28.207740 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:28.207709    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" event={"ID":"72e26efb-19f4-4911-95e3-91a0705615cd","Type":"ContainerDied","Data":"b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887"}
Jul 24 21:56:28.208109 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:28.207704    2522 generic.go:356] "Generic (PLEG): container finished" podID="72e26efb-19f4-4911-95e3-91a0705615cd" containerID="b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887" exitCode=2
Jul 24 21:56:29.593805 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:29.593763    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 24 21:56:29.598665 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:29.598640    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 24 21:56:29.891975 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:29.891903    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" podUID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" containerName="splitter-graph-bce0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:56:29.920167 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:29.920146    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:56:30.062894 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.062843    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e26efb-19f4-4911-95e3-91a0705615cd-kube-rbac-proxy-sar-config\") pod \"72e26efb-19f4-4911-95e3-91a0705615cd\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") "
Jul 24 21:56:30.062894 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.062875    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxfqd\" (UniqueName: \"kubernetes.io/projected/72e26efb-19f4-4911-95e3-91a0705615cd-kube-api-access-vxfqd\") pod \"72e26efb-19f4-4911-95e3-91a0705615cd\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") "
Jul 24 21:56:30.063114 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.062912    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e26efb-19f4-4911-95e3-91a0705615cd-proxy-tls\") pod \"72e26efb-19f4-4911-95e3-91a0705615cd\" (UID: \"72e26efb-19f4-4911-95e3-91a0705615cd\") "
Jul 24 21:56:30.063261 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.063234    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e26efb-19f4-4911-95e3-91a0705615cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "72e26efb-19f4-4911-95e3-91a0705615cd" (UID: "72e26efb-19f4-4911-95e3-91a0705615cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:56:30.065251 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.065223    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e26efb-19f4-4911-95e3-91a0705615cd-kube-api-access-vxfqd" (OuterVolumeSpecName: "kube-api-access-vxfqd") pod "72e26efb-19f4-4911-95e3-91a0705615cd" (UID: "72e26efb-19f4-4911-95e3-91a0705615cd"). InnerVolumeSpecName "kube-api-access-vxfqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:56:30.065364 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.065293    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e26efb-19f4-4911-95e3-91a0705615cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72e26efb-19f4-4911-95e3-91a0705615cd" (UID: "72e26efb-19f4-4911-95e3-91a0705615cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:56:30.164316 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.164261    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e26efb-19f4-4911-95e3-91a0705615cd-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:56:30.164316 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.164282    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e26efb-19f4-4911-95e3-91a0705615cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:56:30.164316 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.164291    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxfqd\" (UniqueName: \"kubernetes.io/projected/72e26efb-19f4-4911-95e3-91a0705615cd-kube-api-access-vxfqd\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:56:30.218088 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.218059    2522 generic.go:356] "Generic (PLEG): container finished" podID="72e26efb-19f4-4911-95e3-91a0705615cd" containerID="27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77" exitCode=0
Jul 24 21:56:30.218183 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.218115    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"
Jul 24 21:56:30.218183 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.218146    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" event={"ID":"72e26efb-19f4-4911-95e3-91a0705615cd","Type":"ContainerDied","Data":"27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77"}
Jul 24 21:56:30.218270 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.218211    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b" event={"ID":"72e26efb-19f4-4911-95e3-91a0705615cd","Type":"ContainerDied","Data":"514685e9aaec6161d19997e37aaa0232ae419a0ba79496bb9afcdfa6fcb5230c"}
Jul 24 21:56:30.218270 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.218239    2522 scope.go:117] "RemoveContainer" containerID="b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887"
Jul 24 21:56:30.227342 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.227327    2522 scope.go:117] "RemoveContainer" containerID="27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77"
Jul 24 21:56:30.234867 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.234849    2522 scope.go:117] "RemoveContainer" containerID="b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887"
Jul 24 21:56:30.235119 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:56:30.235102    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887\": container with ID starting with b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887 not found: ID does not exist" containerID="b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887"
Jul 24 21:56:30.235197 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.235132    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887"} err="failed to get container status \"b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887\": rpc error: code = NotFound desc = could not find container \"b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887\": container with ID starting with b746bc80d659f26d152e7d3fbf1a6e2b72dd0f02691b3c2fae6e3cdcaa85a887 not found: ID does not exist"
Jul 24 21:56:30.235197 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.235156    2522 scope.go:117] "RemoveContainer" containerID="27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77"
Jul 24 21:56:30.235391 ip-10-0-140-162 kubenswrapper[2522]: E0724 21:56:30.235375    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77\": container with ID starting with 27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77 not found: ID does not exist" containerID="27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77"
Jul 24 21:56:30.235435 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.235397    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77"} err="failed to get container status \"27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77\": rpc error: code = NotFound desc = could not find container \"27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77\": container with ID starting with 27e2312a6bdbc3a1a5230e41c763e14ec7003239376514e891180e701b05ff77 not found: ID does not exist"
Jul 24 21:56:30.240122 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.240099    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"]
Jul 24 21:56:30.246183 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.246163    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bce0e-predictor-8fffdbfb9-78z7b"]
Jul 24 21:56:30.406416 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:30.406389    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" path="/var/lib/kubelet/pods/72e26efb-19f4-4911-95e3-91a0705615cd/volumes"
Jul 24 21:56:34.891758 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:34.891715    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" podUID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" containerName="splitter-graph-bce0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:56:39.891557 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:39.891502    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" podUID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" containerName="splitter-graph-bce0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:56:39.892048 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:39.891661    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:56:44.891784 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:44.891739    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" podUID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" containerName="splitter-graph-bce0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:56:49.891233 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:49.891187    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" podUID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" containerName="splitter-graph-bce0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:56:54.892115 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:54.892077    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" podUID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" containerName="splitter-graph-bce0e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:56:57.347023 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:57.346988    2522 generic.go:356] "Generic (PLEG): container finished" podID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" containerID="008254a1056bf60223aae21e5999e7188d8eb56dea6367b617bf9cb6a9ad64cc" exitCode=0
Jul 24 21:56:57.347363 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:57.347033    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" event={"ID":"1c9a9c53-5c0f-42d3-a45b-071d67b40be8","Type":"ContainerDied","Data":"008254a1056bf60223aae21e5999e7188d8eb56dea6367b617bf9cb6a9ad64cc"}
Jul 24 21:56:57.368692 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:57.368649    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:56:57.462332 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:57.462265    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-proxy-tls\") pod \"1c9a9c53-5c0f-42d3-a45b-071d67b40be8\" (UID: \"1c9a9c53-5c0f-42d3-a45b-071d67b40be8\") "
Jul 24 21:56:57.462448 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:57.462334    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-openshift-service-ca-bundle\") pod \"1c9a9c53-5c0f-42d3-a45b-071d67b40be8\" (UID: \"1c9a9c53-5c0f-42d3-a45b-071d67b40be8\") "
Jul 24 21:56:57.462606 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:57.462586    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1c9a9c53-5c0f-42d3-a45b-071d67b40be8" (UID: "1c9a9c53-5c0f-42d3-a45b-071d67b40be8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:56:57.464248 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:57.464228    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c9a9c53-5c0f-42d3-a45b-071d67b40be8" (UID: "1c9a9c53-5c0f-42d3-a45b-071d67b40be8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:56:57.563761 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:57.563736    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:56:57.563761 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:57.563756    2522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c9a9c53-5c0f-42d3-a45b-071d67b40be8-openshift-service-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 21:56:58.353988 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:58.353956    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"
Jul 24 21:56:58.354464 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:58.353999    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh" event={"ID":"1c9a9c53-5c0f-42d3-a45b-071d67b40be8","Type":"ContainerDied","Data":"86c4812d04aeceb8f97e0018b1b0a2745795a6c671758719f043215ded5085ec"}
Jul 24 21:56:58.354464 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:58.354074    2522 scope.go:117] "RemoveContainer" containerID="008254a1056bf60223aae21e5999e7188d8eb56dea6367b617bf9cb6a9ad64cc"
Jul 24 21:56:58.376725 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:58.376700    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"]
Jul 24 21:56:58.380444 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:58.380422    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bce0e-8f65fd6fd-zlngh"]
Jul 24 21:56:58.406403 ip-10-0-140-162 kubenswrapper[2522]: I0724 21:56:58.406381    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" path="/var/lib/kubelet/pods/1c9a9c53-5c0f-42d3-a45b-071d67b40be8/volumes"
Jul 24 22:03:53.515736 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:03:53.515702    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:11.332145 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:11.332048    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"]
Jul 24 22:04:11.332586 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:11.332491    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" podUID="cd9e0c35-68db-42f7-a4a9-066869594ece" containerName="switch-graph-a916f" containerID="cri-o://ede1fd83b1f29268a597cf3102b05dd4e489ed52c164002498b2690ca5c9948c" gracePeriod=30
Jul 24 22:04:11.367500 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:11.367469    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:11.542726 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:11.542694    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:11.573018 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:11.572994    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"]
Jul 24 22:04:11.573239 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:11.573216    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kserve-container" containerID="cri-o://a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec" gracePeriod=30
Jul 24 22:04:11.573301 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:11.573238    2522 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kube-rbac-proxy" containerID="cri-o://d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad" gracePeriod=30
Jul 24 22:04:11.609785 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:11.609759    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:12.019092 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:12.019061    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:12.082341 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:12.082303    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:12.348620 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:12.348527    2522 generic.go:356] "Generic (PLEG): container finished" podID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerID="d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad" exitCode=2
Jul 24 22:04:12.348620 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:12.348574    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" event={"ID":"40d3f802-7324-4f66-b47e-4a3ba18fe0aa","Type":"ContainerDied","Data":"d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad"}
Jul 24 22:04:14.215587 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.215563    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 22:04:14.289443 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.289379    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-rbac-proxy-sar-config\") pod \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") "
Jul 24 22:04:14.289565 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.289487    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz7qr\" (UniqueName: \"kubernetes.io/projected/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-api-access-lz7qr\") pod \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") "
Jul 24 22:04:14.289565 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.289549    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-proxy-tls\") pod \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\" (UID: \"40d3f802-7324-4f66-b47e-4a3ba18fe0aa\") "
Jul 24 22:04:14.289798 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.289775    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40d3f802-7324-4f66-b47e-4a3ba18fe0aa" (UID: "40d3f802-7324-4f66-b47e-4a3ba18fe0aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:04:14.291653 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.291620    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-api-access-lz7qr" (OuterVolumeSpecName: "kube-api-access-lz7qr") pod "40d3f802-7324-4f66-b47e-4a3ba18fe0aa" (UID: "40d3f802-7324-4f66-b47e-4a3ba18fe0aa"). InnerVolumeSpecName "kube-api-access-lz7qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:04:14.291793 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.291657    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40d3f802-7324-4f66-b47e-4a3ba18fe0aa" (UID: "40d3f802-7324-4f66-b47e-4a3ba18fe0aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:04:14.358844 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.358820    2522 generic.go:356] "Generic (PLEG): container finished" podID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerID="a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec" exitCode=0
Jul 24 22:04:14.358940 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.358865    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"
Jul 24 22:04:14.358940 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.358907    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" event={"ID":"40d3f802-7324-4f66-b47e-4a3ba18fe0aa","Type":"ContainerDied","Data":"a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec"}
Jul 24 22:04:14.359014 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.358964    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2" event={"ID":"40d3f802-7324-4f66-b47e-4a3ba18fe0aa","Type":"ContainerDied","Data":"de1b4e24950d13d7b9806b9174fdc04a16eb3f84d9ba8ad26c8a9bb3f2d90ca9"}
Jul 24 22:04:14.359014 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.358992    2522 scope.go:117] "RemoveContainer" containerID="d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad"
Jul 24 22:04:14.367947 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.367933    2522 scope.go:117] "RemoveContainer" containerID="a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec"
Jul 24 22:04:14.375004 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.374990    2522 scope.go:117] "RemoveContainer" containerID="d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad"
Jul 24 22:04:14.375241 ip-10-0-140-162 kubenswrapper[2522]: E0724 22:04:14.375222    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad\": container with ID starting with d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad not found: ID does not exist" containerID="d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad"
Jul 24 22:04:14.375334 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.375247    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad"} err="failed to get container status \"d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad\": rpc error: code = NotFound desc = could not find container \"d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad\": container with ID starting with d8232a179a63a2c43a9f168713c1687e67f930366e4d96f52d900c768673b9ad not found: ID does not exist"
Jul 24 22:04:14.375334 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.375264    2522 scope.go:117] "RemoveContainer" containerID="a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec"
Jul 24 22:04:14.375469 ip-10-0-140-162 kubenswrapper[2522]: E0724 22:04:14.375450    2522 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec\": container with ID starting with a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec not found: ID does not exist" containerID="a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec"
Jul 24 22:04:14.375510 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.375475    2522 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec"} err="failed to get container status \"a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec\": rpc error: code = NotFound desc = could not find container \"a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec\": container with ID starting with a8e1f6b5c392c21403d9809715b843dc6ee69802e1005806833d3e1b7bfac8ec not found: ID does not exist"
Jul 24 22:04:14.380345 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.380321    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"]
Jul 24 22:04:14.383512 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.383490    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a916f-predictor-5548f55f77-vn8q2"]
Jul 24 22:04:14.390091 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.390073    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 22:04:14.390175 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.390095    2522 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 22:04:14.390175 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.390110    2522 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz7qr\" (UniqueName: \"kubernetes.io/projected/40d3f802-7324-4f66-b47e-4a3ba18fe0aa-kube-api-access-lz7qr\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 22:04:14.405831 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:14.405812    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" path="/var/lib/kubelet/pods/40d3f802-7324-4f66-b47e-4a3ba18fe0aa/volumes"
Jul 24 22:04:15.802820 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:15.802771    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" podUID="cd9e0c35-68db-42f7-a4a9-066869594ece" containerName="switch-graph-a916f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 22:04:20.802770 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:20.802725    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" podUID="cd9e0c35-68db-42f7-a4a9-066869594ece" containerName="switch-graph-a916f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 22:04:25.802898 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:25.802862    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" podUID="cd9e0c35-68db-42f7-a4a9-066869594ece" containerName="switch-graph-a916f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 22:04:25.803403 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:25.802963    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 22:04:28.099902 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:28.099862    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:28.925965 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:28.925932    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:29.772189 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:29.772155    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:30.581377 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:30.581345    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:30.805651 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:30.805606    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" podUID="cd9e0c35-68db-42f7-a4a9-066869594ece" containerName="switch-graph-a916f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 22:04:31.393649 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:31.393605    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:32.257853 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:32.257811    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:33.056605 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:33.056573    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:33.881540 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:33.881513    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:34.866779 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:34.866738    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:35.725892 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:35.725850    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:35.803431 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:35.803399    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" podUID="cd9e0c35-68db-42f7-a4a9-066869594ece" containerName="switch-graph-a916f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 22:04:36.568067 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:36.568030    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:37.383228 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:37.383191    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:38.254773 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:38.254741    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:39.074792 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:39.074664    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-a916f-6d9bcff485-5rs4f_cd9e0c35-68db-42f7-a4a9-066869594ece/switch-graph-a916f/0.log"
Jul 24 22:04:40.802930 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:40.802896    2522 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" podUID="cd9e0c35-68db-42f7-a4a9-066869594ece" containerName="switch-graph-a916f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 22:04:41.194733 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.194698    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:41.338076 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.338049    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:41.439809 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.439777    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:41.484990 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.484967    2522 generic.go:356] "Generic (PLEG): container finished" podID="cd9e0c35-68db-42f7-a4a9-066869594ece" containerID="ede1fd83b1f29268a597cf3102b05dd4e489ed52c164002498b2690ca5c9948c" exitCode=0
Jul 24 22:04:41.485103 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.485013    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" event={"ID":"cd9e0c35-68db-42f7-a4a9-066869594ece","Type":"ContainerDied","Data":"ede1fd83b1f29268a597cf3102b05dd4e489ed52c164002498b2690ca5c9948c"}
Jul 24 22:04:41.485103 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.485051    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f" event={"ID":"cd9e0c35-68db-42f7-a4a9-066869594ece","Type":"ContainerDied","Data":"a1721c42df9dd81e7c158a23464d4cef53251d5fb27a16147bd1aa6f49158513"}
Jul 24 22:04:41.485103 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.485068    2522 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1721c42df9dd81e7c158a23464d4cef53251d5fb27a16147bd1aa6f49158513"
Jul 24 22:04:41.491795 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.491777    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 22:04:41.582817 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.582794    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd9e0c35-68db-42f7-a4a9-066869594ece-openshift-service-ca-bundle\") pod \"cd9e0c35-68db-42f7-a4a9-066869594ece\" (UID: \"cd9e0c35-68db-42f7-a4a9-066869594ece\") "
Jul 24 22:04:41.583078 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.582877    2522 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls\") pod \"cd9e0c35-68db-42f7-a4a9-066869594ece\" (UID: \"cd9e0c35-68db-42f7-a4a9-066869594ece\") "
Jul 24 22:04:41.583164 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.583144    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9e0c35-68db-42f7-a4a9-066869594ece-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "cd9e0c35-68db-42f7-a4a9-066869594ece" (UID: "cd9e0c35-68db-42f7-a4a9-066869594ece"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:04:41.584957 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.584932    2522 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd9e0c35-68db-42f7-a4a9-066869594ece" (UID: "cd9e0c35-68db-42f7-a4a9-066869594ece"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:04:41.684387 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.684366    2522 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd9e0c35-68db-42f7-a4a9-066869594ece-openshift-service-ca-bundle\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 22:04:41.684484 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:41.684394    2522 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd9e0c35-68db-42f7-a4a9-066869594ece-proxy-tls\") on node \"ip-10-0-140-162.ec2.internal\" DevicePath \"\""
Jul 24 22:04:42.490266 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:42.490234    2522 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"
Jul 24 22:04:42.504211 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:42.504186    2522 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"]
Jul 24 22:04:42.507973 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:42.507949    2522 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a916f-6d9bcff485-5rs4f"]
Jul 24 22:04:43.941041 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:43.941012    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sxqs9_86301808-2ce5-4813-b527-98b3967b4d15/global-pull-secret-syncer/0.log"
Jul 24 22:04:44.043709 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:44.043661    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log"
Jul 24 22:04:44.185749 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:44.185723    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-162.ec2.internal_66217b0fb27a8d11ff5105a5ffa34535/haproxy/0.log"
Jul 24 22:04:44.407027 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:44.406950    2522 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd9e0c35-68db-42f7-a4a9-066869594ece" path="/var/lib/kubelet/pods/cd9e0c35-68db-42f7-a4a9-066869594ece/volumes"
Jul 24 22:04:46.924877 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:46.924844    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:46.990487 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:46.990459    2522 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gfsnb_6159f1d7-ac4d-4c75-a777-636040b75350/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:48.399635 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.399599    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_69cd2ae2-6450-4396-8ab8-97075bf3978f/alertmanager/0.log"
Jul 24 22:04:48.433295 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.433217    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_69cd2ae2-6450-4396-8ab8-97075bf3978f/config-reloader/0.log"
Jul 24 22:04:48.457988 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.457966    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_69cd2ae2-6450-4396-8ab8-97075bf3978f/kube-rbac-proxy-web/0.log"
Jul 24 22:04:48.487838 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.487815    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_69cd2ae2-6450-4396-8ab8-97075bf3978f/kube-rbac-proxy/0.log"
Jul 24 22:04:48.537197 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.537178    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_69cd2ae2-6450-4396-8ab8-97075bf3978f/kube-rbac-proxy-metric/0.log"
Jul 24 22:04:48.606749 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.606731    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_69cd2ae2-6450-4396-8ab8-97075bf3978f/prom-label-proxy/0.log"
Jul 24 22:04:48.661308 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.661290    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_69cd2ae2-6450-4396-8ab8-97075bf3978f/init-config-reloader/0.log"
Jul 24 22:04:48.712220 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.712201    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-5rdtx_689dd5e0-863e-4f54-af4b-3c268bfe6351/cluster-monitoring-operator/0.log"
Jul 24 22:04:48.740371 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.740350    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-sznnf_0fc70724-9e58-4b11-9c34-3b9563dfd847/kube-state-metrics/0.log"
Jul 24 22:04:48.759956 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.759935    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-sznnf_0fc70724-9e58-4b11-9c34-3b9563dfd847/kube-rbac-proxy-main/0.log"
Jul 24 22:04:48.784646 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.784626    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-sznnf_0fc70724-9e58-4b11-9c34-3b9563dfd847/kube-rbac-proxy-self/0.log"
Jul 24 22:04:48.975309 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:48.975254    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b4zps_e5ea6b64-385e-40a4-84fb-bc4666e8c2a3/node-exporter/0.log"
Jul 24 22:04:49.000763 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:49.000723    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b4zps_e5ea6b64-385e-40a4-84fb-bc4666e8c2a3/kube-rbac-proxy/0.log"
Jul 24 22:04:49.028635 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:49.028614    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b4zps_e5ea6b64-385e-40a4-84fb-bc4666e8c2a3/init-textfile/0.log"
Jul 24 22:04:49.131330 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:49.131310    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-drrpz_0a96bf67-2b22-4d4d-8486-3f01d422d89b/kube-rbac-proxy-main/0.log"
Jul 24 22:04:49.157843 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:49.157825    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-drrpz_0a96bf67-2b22-4d4d-8486-3f01d422d89b/kube-rbac-proxy-self/0.log"
Jul 24 22:04:49.188553 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:49.188534    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-drrpz_0a96bf67-2b22-4d4d-8486-3f01d422d89b/openshift-state-metrics/0.log"
Jul 24 22:04:49.421929 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:49.421844    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-74kgg_d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b/prometheus-operator/0.log"
Jul 24 22:04:49.456066 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:49.456035    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-74kgg_d6299c5f-caa4-4b2d-ac55-a6d0c50ac34b/kube-rbac-proxy/0.log"
Jul 24 22:04:49.507897 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:49.507866    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-hrnqd_c07971e1-0f44-4fcd-8331-c5efe52c3fc1/prometheus-operator-admission-webhook/0.log"
Jul 24 22:04:52.040455 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040417    2522 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"]
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040769    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kserve-container"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040782    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kserve-container"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040804    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kube-rbac-proxy"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040810    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kube-rbac-proxy"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040818    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd9e0c35-68db-42f7-a4a9-066869594ece" containerName="switch-graph-a916f"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040823    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd9e0c35-68db-42f7-a4a9-066869594ece" containerName="switch-graph-a916f"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040833    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kube-rbac-proxy"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040838    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kube-rbac-proxy"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040845    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kserve-container"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040851    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kserve-container"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040858    2522 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" containerName="splitter-graph-bce0e"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040863    2522 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" containerName="splitter-graph-bce0e"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040916    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd9e0c35-68db-42f7-a4a9-066869594ece" containerName="switch-graph-a916f"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040924    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kserve-container"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040932    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c9a9c53-5c0f-42d3-a45b-071d67b40be8" containerName="splitter-graph-bce0e"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040938    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kube-rbac-proxy"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040944    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d3f802-7324-4f66-b47e-4a3ba18fe0aa" containerName="kube-rbac-proxy"
Jul 24 22:04:52.040943 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.040950    2522 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e26efb-19f4-4911-95e3-91a0705615cd" containerName="kserve-container"
Jul 24 22:04:52.044062 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.044040    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.046859 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.046828    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9x45t\"/\"openshift-service-ca.crt\""
Jul 24 22:04:52.047012 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.046886    2522 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9x45t\"/\"kube-root-ca.crt\""
Jul 24 22:04:52.047012 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.046967    2522 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9x45t\"/\"default-dockercfg-4m76b\""
Jul 24 22:04:52.057528 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.057504    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"]
Jul 24 22:04:52.157664 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.157632    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-lib-modules\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.157824 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.157695    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-podres\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.157824 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.157754    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbmkj\" (UniqueName: \"kubernetes.io/projected/02268beb-596e-44cb-a5a5-b54433ae7db2-kube-api-access-fbmkj\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.157824 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.157785    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-sys\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.157824 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.157806    2522 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-proc\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.258656 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.258631    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbmkj\" (UniqueName: \"kubernetes.io/projected/02268beb-596e-44cb-a5a5-b54433ae7db2-kube-api-access-fbmkj\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.258795 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.258659    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-sys\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.258795 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.258697    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-proc\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.258795 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.258749    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-lib-modules\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.258795 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.258764    2522 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-podres\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.258795 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.258779    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-sys\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.259006 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.258846    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-proc\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.259006 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.258883    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-podres\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.259006 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.258904    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02268beb-596e-44cb-a5a5-b54433ae7db2-lib-modules\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.267615 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.267593    2522 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbmkj\" (UniqueName: \"kubernetes.io/projected/02268beb-596e-44cb-a5a5-b54433ae7db2-kube-api-access-fbmkj\") pod \"perf-node-gather-daemonset-zmwjj\" (UID: \"02268beb-596e-44cb-a5a5-b54433ae7db2\") " pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.355197 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.355133    2522 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:52.485588 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.485545    2522 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"]
Jul 24 22:04:52.488357 ip-10-0-140-162 kubenswrapper[2522]: W0724 22:04:52.488288    2522 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod02268beb_596e_44cb_a5a5_b54433ae7db2.slice/crio-5bb55f6f0051ee148f7b7524c43be6bcd288a6de0d404068a8fdced3df885898 WatchSource:0}: Error finding container 5bb55f6f0051ee148f7b7524c43be6bcd288a6de0d404068a8fdced3df885898: Status 404 returned error can't find the container with id 5bb55f6f0051ee148f7b7524c43be6bcd288a6de0d404068a8fdced3df885898
Jul 24 22:04:52.534031 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:52.534008    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj" event={"ID":"02268beb-596e-44cb-a5a5-b54433ae7db2","Type":"ContainerStarted","Data":"5bb55f6f0051ee148f7b7524c43be6bcd288a6de0d404068a8fdced3df885898"}
Jul 24 22:04:53.316250 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:53.316218    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-krwsr_1e21e54d-fb5e-4a9a-9f64-908449a5e755/dns/0.log"
Jul 24 22:04:53.336855 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:53.336832    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-krwsr_1e21e54d-fb5e-4a9a-9f64-908449a5e755/kube-rbac-proxy/0.log"
Jul 24 22:04:53.406563 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:53.406538    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sw9pw_09ad0063-0c05-496f-9696-cec68aa2aa14/dns-node-resolver/0.log"
Jul 24 22:04:53.539965 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:53.539939    2522 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj" event={"ID":"02268beb-596e-44cb-a5a5-b54433ae7db2","Type":"ContainerStarted","Data":"8d69717956040ac4a82692d086b8dad3393d89f2242f80515912c23c43f24645"}
Jul 24 22:04:53.540111 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:53.540004    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:04:53.557493 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:53.557452    2522 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj" podStartSLOduration=1.557440053 podStartE2EDuration="1.557440053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:04:53.555701247 +0000 UTC m=+3015.587096258" watchObservedRunningTime="2026-07-24 22:04:53.557440053 +0000 UTC m=+3015.588835060"
Jul 24 22:04:53.925778 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:53.925756    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zsccb_b7a9829f-33e2-4b02-a05a-35f9765d9fbb/node-ca/0.log"
Jul 24 22:04:54.623794 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:54.623766    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-749656c945-wnf2b_99a5f181-09a0-4f61-85a7-6946878f7980/router/0.log"
Jul 24 22:04:54.966616 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:54.966585    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j7h2p_7ff7824c-d44f-48f1-af97-518f5b3b5d9b/serve-healthcheck-canary/0.log"
Jul 24 22:04:55.470442 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:55.470417    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sqv7q_043caca9-caf7-46cb-9297-b7be1f5fd9a9/kube-rbac-proxy/0.log"
Jul 24 22:04:55.494911 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:55.494883    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sqv7q_043caca9-caf7-46cb-9297-b7be1f5fd9a9/exporter/0.log"
Jul 24 22:04:55.518814 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:55.518788    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sqv7q_043caca9-caf7-46cb-9297-b7be1f5fd9a9/extractor/0.log"
Jul 24 22:04:57.794466 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:57.794442    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-664687df58-584jx_2f0a1a16-9f43-4cad-96f6-9ba84516abdf/manager/0.log"
Jul 24 22:04:57.829017 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:57.828994    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-nwfkz_e31f3f1c-99c8-4d58-bc93-2b6b8521e2d6/manager/0.log"
Jul 24 22:04:59.555031 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:04:59.554998    2522 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9x45t/perf-node-gather-daemonset-zmwjj"
Jul 24 22:05:03.356455 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:03.356430    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcw5h_82e50870-99f4-4b0c-91ad-dbfca23e5ca7/kube-multus-additional-cni-plugins/0.log"
Jul 24 22:05:03.378751 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:03.378730    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcw5h_82e50870-99f4-4b0c-91ad-dbfca23e5ca7/egress-router-binary-copy/0.log"
Jul 24 22:05:03.401854 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:03.401828    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcw5h_82e50870-99f4-4b0c-91ad-dbfca23e5ca7/cni-plugins/0.log"
Jul 24 22:05:03.423047 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:03.423028    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcw5h_82e50870-99f4-4b0c-91ad-dbfca23e5ca7/bond-cni-plugin/0.log"
Jul 24 22:05:03.442825 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:03.442805    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcw5h_82e50870-99f4-4b0c-91ad-dbfca23e5ca7/routeoverride-cni/0.log"
Jul 24 22:05:03.463725 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:03.463701    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcw5h_82e50870-99f4-4b0c-91ad-dbfca23e5ca7/whereabouts-cni-bincopy/0.log"
Jul 24 22:05:03.485305 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:03.485281    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xcw5h_82e50870-99f4-4b0c-91ad-dbfca23e5ca7/whereabouts-cni/0.log"
Jul 24 22:05:03.714051 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:03.714020    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r57zx_523501ee-3e85-408e-9dac-a8cee654bc6a/kube-multus/0.log"
Jul 24 22:05:03.794608 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:03.794577    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-krmvj_fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9/network-metrics-daemon/0.log"
Jul 24 22:05:03.816788 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:03.816771    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-krmvj_fcc4cf5b-9c84-4c3d-8009-edcb10e69ca9/kube-rbac-proxy/0.log"
Jul 24 22:05:04.685039 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:04.685012    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntvd4_6f320b45-0781-45f5-916d-5ac2fe9fa951/ovn-controller/0.log"
Jul 24 22:05:04.701636 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:04.701617    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntvd4_6f320b45-0781-45f5-916d-5ac2fe9fa951/ovn-acl-logging/0.log"
Jul 24 22:05:04.727260 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:04.727203    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntvd4_6f320b45-0781-45f5-916d-5ac2fe9fa951/kube-rbac-proxy-node/0.log"
Jul 24 22:05:04.747843 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:04.747821    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntvd4_6f320b45-0781-45f5-916d-5ac2fe9fa951/kube-rbac-proxy-ovn-metrics/0.log"
Jul 24 22:05:04.766435 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:04.766415    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntvd4_6f320b45-0781-45f5-916d-5ac2fe9fa951/northd/0.log"
Jul 24 22:05:04.785763 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:04.785738    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntvd4_6f320b45-0781-45f5-916d-5ac2fe9fa951/nbdb/0.log"
Jul 24 22:05:04.806355 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:04.806332    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntvd4_6f320b45-0781-45f5-916d-5ac2fe9fa951/sbdb/0.log"
Jul 24 22:05:04.873345 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:04.873322    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntvd4_6f320b45-0781-45f5-916d-5ac2fe9fa951/ovnkube-controller/0.log"
Jul 24 22:05:06.305932 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:06.305902    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-bsshq_27b45efa-aab7-4a1f-a6fb-3a701fdb7d89/check-endpoints/0.log"
Jul 24 22:05:06.326852 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:06.326823    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9nzl9_05180a7b-cf89-498d-af4b-6ba6ed49ae0e/network-check-target-container/0.log"
Jul 24 22:05:07.297487 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:07.297452    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xlrn4_2c62472f-b8c0-4ea0-9d99-5703ddf06036/iptables-alerter/0.log"
Jul 24 22:05:07.958242 ip-10-0-140-162 kubenswrapper[2522]: I0724 22:05:07.958214    2522 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-xdzf5_9970fe5c-3f27-42a5-adca-5f2fb3b3e935/tuned/0.log"