Jul 02 13:44:22.654416 ip-10-0-138-214 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 13:44:23.169986 ip-10-0-138-214 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:44:23.169986 ip-10-0-138-214 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 13:44:23.169986 ip-10-0-138-214 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:44:23.169986 ip-10-0-138-214 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 13:44:23.169986 ip-10-0-138-214 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:44:23.170700 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.170057    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 13:44:23.178841 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178821    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:44:23.178841 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178839    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:44:23.178841 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178844    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178848    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178852    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178855    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178858    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178861    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178864    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178867    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178870    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178873    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178876    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178879    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178888    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178891    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178894    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178897    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178899    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178902    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178905    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178908    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:44:23.178943 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178912    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178914    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178917    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178920    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178923    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178926    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178928    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178931    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178934    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178937    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178940    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178943    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178946    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178949    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178953    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178956    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178961    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178963    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178966    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178969    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:44:23.179584 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178972    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178974    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178977    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178980    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178982    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178986    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178989    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178992    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178995    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.178998    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179000    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179004    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179006    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179009    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179013    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179018    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179024    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179027    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179030    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:44:23.180138 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179033    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179038    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179041    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179044    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179047    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179050    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179053    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179056    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179059    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179062    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179064    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179067    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179070    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179073    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179075    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179078    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179081    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179084    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179088    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179091    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:44:23.180650 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179094    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179098    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179101    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179104    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179107    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179110    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179113    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179117    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179121    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179126    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179130    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179133    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179136    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179139    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179142    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179145    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179148    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179151    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179982    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:44:23.181164 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179990    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179994    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.179997    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180002    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180005    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180008    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180011    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180014    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180017    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180020    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180023    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180025    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180028    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180032    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180037    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180041    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180044    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180047    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180050    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:44:23.181668 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180053    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180056    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180060    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180062    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180065    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180068    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180071    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180074    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180076    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180080    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180082    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180085    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180088    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180091    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180094    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180097    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180101    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180104    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180106    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:44:23.182163 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180109    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180112    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180115    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180118    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180121    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180124    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180127    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180130    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180133    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180136    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180139    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180142    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180145    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180148    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180152    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180156    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180158    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180161    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180164    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180168    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:44:23.182689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180170    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180173    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180176    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180180    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180184    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180187    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180192    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180195    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180198    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180200    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180203    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180206    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180228    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180231    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180234    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180237    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180241    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180245    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180248    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180251    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:44:23.183207 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180255    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180258    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180261    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180264    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180267    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180269    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180273    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180276    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180279    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180282    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180285    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180288    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180290    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180293    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180296    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180298    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180301    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180304    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180307    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180310    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:44:23.183732 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180808    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180814    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180818    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180821    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180825    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180828    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180832    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180836    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180839    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180882    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180890    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180894    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180898    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180901    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180905    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180908    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180912    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180914    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180918    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180921    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:44:23.184276 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180924    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180927    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180930    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180933    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180938    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180941    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180947    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180950    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180955    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180960    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180963    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180967    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180970    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180973    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180976    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180979    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180982    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180985    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180988    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:44:23.184801 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180992    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.180996    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181000    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181004    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181007    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181011    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181014    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181017    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181020    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181023    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181026    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181028    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181032    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181035    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181038    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181040    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181043    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181046    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181050    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181053    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:44:23.185316 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181056    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181058    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181061    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181066    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181068    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181071    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181074    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181077    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181079    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181090    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181094    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181097    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181099    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181102    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181106    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181110    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181112    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181115    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181118    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:44:23.185839 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181121    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181124    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181127    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181130    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181132    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181135    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181138    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181142    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181146    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181149    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181152    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181155    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181158    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181161    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181163    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181166    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181169    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181172    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181175    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181177    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:44:23.186366 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181180    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181281    2574 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181291    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181300    2574 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181305    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181310    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181313    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181320    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181325    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181329    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181332    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181336    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181340    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181343    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181346    2574 flags.go:64] FLAG: --cgroup-root=""
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181350    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181353    2574 flags.go:64] FLAG: --client-ca-file=""
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181356    2574 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181359    2574 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181364    2574 flags.go:64] FLAG: --cluster-domain=""
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181367    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181370    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181374    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181378    2574 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 13:44:23.186882 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181382    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181386    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181389    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181393    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181396    2574 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181399    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181402    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181406    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181409    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181413    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181416    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181419    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181423    2574 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181426    2574 flags.go:64] FLAG: --enable-server="true"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181429    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181436    2574 flags.go:64] FLAG: --event-burst="100"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181440    2574 flags.go:64] FLAG: --event-qps="50"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181443    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181446    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181450    2574 flags.go:64] FLAG: --eviction-hard=""
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181454    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181457    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181460    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181463    2574 flags.go:64] FLAG: --eviction-soft=""
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181466    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 13:44:23.187520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181469    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181472    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181475    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181479    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181482    2574 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181485    2574 flags.go:64] FLAG: --feature-gates=""
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181489    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181492    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181495    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181499    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181502    2574 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181505    2574 flags.go:64] FLAG: --help="false"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181508    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181512    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181515    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181518    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181521    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181525    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181528    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181531    2574 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181534    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181537    2574 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181541    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181544    2574 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 13:44:23.188153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181547    2574 flags.go:64] FLAG: --kube-reserved=""
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181550    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181553    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181556    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181559    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181563    2574 flags.go:64] FLAG: --lock-file=""
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181565    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181568    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181572    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181577    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181580    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181583    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181586    2574 flags.go:64] FLAG: --logging-format="text"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181589    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181593    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181596    2574 flags.go:64] FLAG: --manifest-url=""
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181599    2574 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181604    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181607    2574 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181612    2574 flags.go:64] FLAG: --max-pods="110"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181618    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181621    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181624    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181627    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181631    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 13:44:23.188853 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181634    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181637    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181645    2574 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181648    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181651    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181655    2574 flags.go:64] FLAG: --pod-cidr=""
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181658    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181664    2574 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181667    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181671    2574 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181674    2574 flags.go:64] FLAG: --port="10250"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181677    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181681    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00d3adf5f73c3f8e6"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181684    2574 flags.go:64] FLAG: --qos-reserved=""
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181687    2574 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181690    2574 flags.go:64] FLAG: --register-node="true"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181693    2574 flags.go:64] FLAG: --register-with-taints=""
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181697    2574 flags.go:64] FLAG: --registry-burst="10"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181700    2574 flags.go:64] FLAG: --registry-qps="5"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181703    2574 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181705    2574 flags.go:64] FLAG: --reserved-memory=""
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181714    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181717    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181720    2574 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181723    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 13:44:23.189838 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181726    2574 flags.go:64] FLAG: --runonce="false"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181730    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181735    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181738    2574 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181742    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181745    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181748    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181751    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181754    2574 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181757    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181760    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181763    2574 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181767    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181770    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181773    2574 flags.go:64] FLAG: --system-cgroups=""
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181776    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181783    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181786    2574 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181790    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181794    2574 flags.go:64] FLAG: --tls-min-version=""
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181797    2574 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181800    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181803    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181806    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181809    2574 flags.go:64] FLAG: --v="2"
Jul 02 13:44:23.190624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181813    2574 flags.go:64] FLAG: --version="false"
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181817    2574 flags.go:64] FLAG: --vmodule=""
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181821    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.181825    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181957    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181961    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181964    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181967    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181971    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181975    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181978    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181981    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181983    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181986    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181989    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181992    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181995    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.181998    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182001    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182007    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:44:23.191392 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182010    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182013    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182016    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182019    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182022    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182025    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182028    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182030    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182033    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182036    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182039    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182041    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182044    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182047    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182050    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182053    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182055    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182058    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182061    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182064    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:44:23.191946 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182066    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182074    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182077    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182080    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182083    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182086    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182088    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182092    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182096    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182098    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182101    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182106    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182109    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182112    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182116    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182120    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182123    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182126    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182129    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:44:23.192562 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182133    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182136    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182138    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182141    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182144    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182147    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182150    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182152    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182155    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182158    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182161    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182163    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182167    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182170    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182175    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182178    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182181    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182184    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182187    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182189    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:44:23.193060 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182192    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182195    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182198    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182201    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182205    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182223    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182226    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182231    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182234    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182237    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182240    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182243    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182246    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182249    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182252    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182255    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182258    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182260    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182263    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182266    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:44:23.193651 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182269    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182271    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182274    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.182277    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.183023    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.191608    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.191625    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.191644    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.191654    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191726    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191731    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191735    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191738    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191741    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191744    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:44:23.194179 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191747    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191750    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191753    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191755    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191758    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191761    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191764    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191767    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191769    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191772    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191775    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191780    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191783    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191787    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191789    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191792    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191795    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191798    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191802    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191805    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:44:23.194587 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191807    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191810    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191813    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191816    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191819    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191822    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191825    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191828    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191831    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191834    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191836    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191839    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191842    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191845    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191848    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191852    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191856    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191860    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191862    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:44:23.195098 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191865    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191868    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191871    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191874    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191877    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191880    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191883    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191886    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191889    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191891    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191894    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191897    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191899    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191902    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191905    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191908    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191911    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191915    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191918    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191920    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:44:23.195595 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191923    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191926    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191929    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191932    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191934    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191937    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191940    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191942    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191945    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191948    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191951    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191954    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191957    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191960    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191964    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191967    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191969    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191972    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191976    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:44:23.196109 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191980    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191983    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191987    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191990    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191993    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.191996    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192000    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192003    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192006    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192010    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192013    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192015    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192018    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192021    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192024    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:44:23.196634 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.192029    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192168    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192174    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192177    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192180    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192183    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192186    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192189    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192191    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192194    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192197    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192200    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192202    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192206    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192225    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192230    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192234    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192237    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192240    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192243    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:44:23.197049 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192246    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192249    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192251    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192254    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192257    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192260    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192263    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192266    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192269    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192271    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192274    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192277    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192281    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192285    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192289    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192292    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192295    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192298    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192301    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192304    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:44:23.197644 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192306    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192309    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192312    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192315    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192317    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192320    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192324    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192328    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192331    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192334    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192337    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192340    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192343    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192346    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192349    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192352    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192354    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192357    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192360    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:44:23.198161 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192363    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192366    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192368    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192371    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192374    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192377    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192379    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192382    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192386    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192390    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192393    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192395    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192399    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192401    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192404    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192407    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192410    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192412    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192415    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:44:23.198686 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192418    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192422    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192425    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192427    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192430    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192433    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192436    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192438    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192441    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192444    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192446    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192449    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192452    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192455    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192457    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192460    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192463    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192466    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192468    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192471    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:44:23.199180 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192474    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:44:23.199737 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:23.192476    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:44:23.199737 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.192482    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:44:23.199737 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.192598    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 13:44:23.199737 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.196257    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 13:44:23.199737 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.198144    2574 server.go:1019] "Starting client certificate rotation"
Jul 02 13:44:23.199737 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.198258    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:44:23.199737 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.198303    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:44:23.202023 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.202008    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:44:23.203354 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.203337    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:44:23.216120 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.216102    2574 log.go:25] "Validated CRI v1 runtime API"
Jul 02 13:44:23.218718 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.218702    2574 log.go:25] "Validated CRI v1 image API"
Jul 02 13:44:23.219627 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.219614    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 13:44:23.223824 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.223796    2574 fs.go:135] Filesystem UUIDs: map[620e1cf7-520b-4789-b7d3-302b22592704:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c11bd3f4-2235-4fb3-988c-a081a0b26671:/dev/nvme0n1p3]
Jul 02 13:44:23.223920 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.223822    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 13:44:23.230971 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.230950    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:44:23.231352 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.231231    2574 manager.go:216] Machine: {Timestamp:2026-07-02 13:44:23.229071947 +0000 UTC m=+0.424695469 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3127084 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d649d527e12bb760995977958b94a SystemUUID:ec2d649d-527e-12bb-7609-95977958b94a BootID:2f6e1047-3f46-4856-8ab1-f02e89ccff91 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5c:76:f6:1f:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5c:76:f6:1f:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:d9:2f:be:35:d9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 13:44:23.231352 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.231345    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 13:44:23.231499 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.231487    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 13:44:23.235524 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.235497    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 13:44:23.235698 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.235527    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-214.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 13:44:23.236510 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.236497    2574 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 13:44:23.236584 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.236517    2574 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 13:44:23.236584 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.236530    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:44:23.236584 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.236551    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:44:23.238144 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.238131    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 13:44:23.241867 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.241851    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:44:23.242022 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.242010    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 13:44:23.242082 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.242054    2574 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 13:44:23.242082 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.242068    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 13:44:23.242184 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.242091    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 13:44:23.242184 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.242106    2574 kubelet.go:398] "Adding apiserver pod source"
Jul 02 13:44:23.242184 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.242131    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 13:44:23.242545 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.242530    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:44:23.242615 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.242555    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:44:23.245193 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.245173    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 13:44:23.247090 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.247076    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 13:44:23.247137 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.247109    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 13:44:23.248554 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248540    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 13:44:23.248554 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248558    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 13:44:23.248660 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248565    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 13:44:23.248660 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248571    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 13:44:23.248660 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248583    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 13:44:23.248660 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248589    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 13:44:23.248660 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248599    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 13:44:23.248660 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248605    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 13:44:23.248660 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248613    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 13:44:23.248660 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248620    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 13:44:23.248660 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248630    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 13:44:23.248660 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.248637    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 13:44:23.249540 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.249529    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 13:44:23.249540 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.249541    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 13:44:23.252807 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.252791    2574 server.go:1268] "Started kubelet"
Jul 02 13:44:23.253068 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.252891    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 13:44:23.253721 ip-10-0-138-214 systemd[1]: Started Kubernetes Kubelet.
Jul 02 13:44:23.253926 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.253843    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 13:44:23.254042 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.253945    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 13:44:23.255191 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.255131    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-214.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 13:44:23.255191 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.255179    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 13:44:23.255331 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.255187    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-214.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 13:44:23.255387 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.255326    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 13:44:23.255592 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.255577    2574 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 13:44:23.259145 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.259121    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 13:44:23.263676 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.263657    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 13:44:23.263676 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.263671    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 13:44:23.263827 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.262644    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-214.ec2.internal.18be7cedcbc046f1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-214.ec2.internal,UID:ip-10-0-138-214.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-214.ec2.internal,},FirstTimestamp:2026-07-02 13:44:23.252764401 +0000 UTC m=+0.448387908,LastTimestamp:2026-07-02 13:44:23.252764401 +0000 UTC m=+0.448387908,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-214.ec2.internal,}"
Jul 02 13:44:23.263827 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.263740    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:23.264632 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.264539    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 13:44:23.264632 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.264561    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 13:44:23.264759 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.264660    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 13:44:23.264759 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.264686    2574 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 13:44:23.264759 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.264695    2574 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 13:44:23.265042 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.265023    2574 factory.go:55] Registering systemd factory
Jul 02 13:44:23.265094 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.265046    2574 factory.go:223] Registration of the systemd container factory successfully
Jul 02 13:44:23.266721 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.266680    2574 factory.go:153] Registering CRI-O factory
Jul 02 13:44:23.266827 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.266749    2574 factory.go:223] Registration of the crio container factory successfully
Jul 02 13:44:23.266827 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.266817    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 13:44:23.266958 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.266842    2574 factory.go:103] Registering Raw factory
Jul 02 13:44:23.266958 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.266859    2574 manager.go:1196] Started watching for new ooms in manager
Jul 02 13:44:23.267555 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.267525    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 13:44:23.267555 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.267530    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-214.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 13:44:23.267947 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.267922    2574 manager.go:349] Starting recovery of all containers
Jul 02 13:44:23.274753 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.274726    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rs4bc"
Jul 02 13:44:23.281338 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.281317    2574 manager.go:354] Recovery completed
Jul 02 13:44:23.282962 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.282941    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rs4bc"
Jul 02 13:44:23.285934 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.285921    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:44:23.288017 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.288000    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:44:23.288073 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.288032    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:44:23.288073 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.288043    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:44:23.288549 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.288534    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 13:44:23.288549 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.288546    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 13:44:23.288655 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.288565    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:44:23.288655 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.288549    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:44:23.289362 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.289346    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:44:23.289456 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.289377    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:44:23.289456 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.289388    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:44:23.290074 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.290051    2574 policy_none.go:49] "None policy: Start"
Jul 02 13:44:23.290243 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.290079    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 13:44:23.290243 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.290096    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 13:44:23.290243 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.290081    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-214.ec2.internal.18be7cedcdda323a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-214.ec2.internal,UID:ip-10-0-138-214.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-214.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-214.ec2.internal,},FirstTimestamp:2026-07-02 13:44:23.288017466 +0000 UTC m=+0.483640967,LastTimestamp:2026-07-02 13:44:23.288017466 +0000 UTC m=+0.483640967,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-214.ec2.internal,}"
Jul 02 13:44:23.292029 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.292015    2574 policy_none.go:47] "Start"
Jul 02 13:44:23.332850 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.332835    2574 manager.go:337] "Starting Device Plugin manager"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.332869    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.332879    2574 server.go:87] "Starting device plugin registration server"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.333164    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.333208    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.333319    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.333406    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.333415    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.333833    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.333911    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.343178    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.344452    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.344490    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.344514    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.344555    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 13:44:23.359080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.347000    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:44:23.438293 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.438213    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:44:23.439272 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.439254    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:44:23.439384 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.439284    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:44:23.439384 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.439295    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:44:23.439384 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.439319    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.445061 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.445035    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-214.ec2.internal"]
Jul 02 13:44:23.445185 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.445106    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:44:23.446700 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.446684    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:44:23.446774 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.446713    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:44:23.446774 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.446724    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:44:23.448176 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.448163    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:44:23.448336 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.448310    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.448377 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.448344    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:44:23.448600 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.448582    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.448638 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.448615    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-214.ec2.internal\": node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:23.448931 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.448914    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:44:23.448999 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.448938    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:44:23.448999 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.448948    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:44:23.448999 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.448920    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:44:23.448999 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.448973    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:44:23.448999 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.448985    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:44:23.450373 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.450355    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.450472 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.450379    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:44:23.451056 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.451040    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:44:23.451134 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.451066    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:44:23.451134 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.451079    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:44:23.465179 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.465128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d47d563e7b550e551dc66f6bd2f8408e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-214.ec2.internal\" (UID: \"d47d563e7b550e551dc66f6bd2f8408e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.465285 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.465184    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c08fb0291bc2eef1de979b2d2e6320f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal\" (UID: \"c08fb0291bc2eef1de979b2d2e6320f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.465285 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.465217    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c08fb0291bc2eef1de979b2d2e6320f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal\" (UID: \"c08fb0291bc2eef1de979b2d2e6320f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.477361 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.477336    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-214.ec2.internal\" not found" node="ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.482343 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.482324    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-214.ec2.internal\" not found" node="ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.491528 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.491512    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:23.565762 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.565733    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d47d563e7b550e551dc66f6bd2f8408e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-214.ec2.internal\" (UID: \"d47d563e7b550e551dc66f6bd2f8408e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.565919 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.565771    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c08fb0291bc2eef1de979b2d2e6320f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal\" (UID: \"c08fb0291bc2eef1de979b2d2e6320f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.565919 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.565789    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c08fb0291bc2eef1de979b2d2e6320f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal\" (UID: \"c08fb0291bc2eef1de979b2d2e6320f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.565919 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.565772    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d47d563e7b550e551dc66f6bd2f8408e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-214.ec2.internal\" (UID: \"d47d563e7b550e551dc66f6bd2f8408e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.565919 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.565814    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c08fb0291bc2eef1de979b2d2e6320f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal\" (UID: \"c08fb0291bc2eef1de979b2d2e6320f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.565919 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.565793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c08fb0291bc2eef1de979b2d2e6320f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal\" (UID: \"c08fb0291bc2eef1de979b2d2e6320f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.592227 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.592201    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:23.692806 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.692722    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:23.782477 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.782440    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.787131 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:23.787113    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:23.793344 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.793325    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:23.894958 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.894923    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:23.995047 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:23.994973    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:24.097353 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:24.097314    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:24.114444 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.114418    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:44:24.198183 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:24.198152    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:24.198183 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.198178    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 13:44:24.198671 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.198345    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 13:44:24.198671 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.198357    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 13:44:24.288411 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.288175    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 13:39:23 +0000 UTC" deadline="2028-01-16 20:54:26.84781896 +0000 UTC"
Jul 02 13:44:24.288411 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.288406    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13519h10m2.559420307s"
Jul 02 13:44:24.298454 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:24.298432    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:24.352056 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:24.352023    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08fb0291bc2eef1de979b2d2e6320f7.slice/crio-5d369e5cc5e92371f40cf6e8542102c15affbceffe451ac4dcb5aa3700b2db63 WatchSource:0}: Error finding container 5d369e5cc5e92371f40cf6e8542102c15affbceffe451ac4dcb5aa3700b2db63: Status 404 returned error can't find the container with id 5d369e5cc5e92371f40cf6e8542102c15affbceffe451ac4dcb5aa3700b2db63
Jul 02 13:44:24.356339 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.356325    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:44:24.375257 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.375229    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:44:24.399099 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:24.399065    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:24.421794 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:24.421771    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47d563e7b550e551dc66f6bd2f8408e.slice/crio-e6fd68e2413ae75a9424fba7c49fdb68c8744e45576c8f021f7b9e9fe09d4855 WatchSource:0}: Error finding container e6fd68e2413ae75a9424fba7c49fdb68c8744e45576c8f021f7b9e9fe09d4855: Status 404 returned error can't find the container with id e6fd68e2413ae75a9424fba7c49fdb68c8744e45576c8f021f7b9e9fe09d4855
Jul 02 13:44:24.501000 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:24.500968    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:24.602718 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:24.602654    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:24.703854 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:24.703823    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-214.ec2.internal\" not found"
Jul 02 13:44:24.711033 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.711013    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:44:24.765150 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.765118    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:24.776238 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.776214    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:44:24.777434 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.777416    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal"
Jul 02 13:44:24.788850 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:24.788830    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:44:25.072269 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.072237    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:44:25.243829 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.243692    2574 apiserver.go:52] "Watching apiserver"
Jul 02 13:44:25.251946 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.251918    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 13:44:25.252382 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.252355    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-wvh8f","openshift-multus/multus-nqrn8","openshift-multus/network-metrics-daemon-6vhv6","openshift-network-operator/iptables-alerter-tmnd6","kube-system/kube-apiserver-proxy-ip-10-0-138-214.ec2.internal","openshift-cluster-node-tuning-operator/tuned-lw6s2","openshift-image-registry/node-ca-d4gsb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal","openshift-network-diagnostics/network-check-target-tlx9j","openshift-ovn-kubernetes/ovnkube-node-lmvr2","kube-system/global-pull-secret-syncer-h9rfz","kube-system/konnectivity-agent-rthpc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6","openshift-dns/node-resolver-sdckb"]
Jul 02 13:44:25.254223 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.254201    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d4gsb"
Jul 02 13:44:25.255932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.255905    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:25.256054 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.255991    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:25.257065 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.257046    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4tz9n\""
Jul 02 13:44:25.257441 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.257420    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 13:44:25.257541 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.257419    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 13:44:25.257724 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.257707    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 13:44:25.258080 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.257971    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:25.258080 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.258035    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:25.259412 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.259387    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tmnd6"
Jul 02 13:44:25.261266 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.260808    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sdckb"
Jul 02 13:44:25.262072 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.262052    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 13:44:25.262189 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.262062    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 13:44:25.262189 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.262089    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:44:25.262307 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.262187    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gkj8m\""
Jul 02 13:44:25.262515 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.262496    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.263638 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.263491    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vrrgz\""
Jul 02 13:44:25.263638 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.263531    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 13:44:25.263638 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.263540    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 13:44:25.263992 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.263965    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 13:44:25.264228 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.264194    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:25.264357 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.264255    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:25.265030 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.265008    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6dxlx\""
Jul 02 13:44:25.265249 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.265232    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:44:25.266102 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.265570    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 13:44:25.266167 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.266103    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.267780 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.267758    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.268578 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.268404    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 13:44:25.268578 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.268566    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pqvtt\""
Jul 02 13:44:25.268876 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.268855    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 13:44:25.269066 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.268861    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 13:44:25.269794 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.269724    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.270561 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.270426    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 13:44:25.270561 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.270427    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 13:44:25.271374 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.271351    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 13:44:25.271587 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.271274    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lztbz\""
Jul 02 13:44:25.271660 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.271607    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 13:44:25.272453 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.272430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.273426 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.273402    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 13:44:25.274051 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.273877    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 13:44:25.275392 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275371    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-registration-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.275498 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275407    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.275498 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275441    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-sys-fs\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.275498 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275473    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9jdc\" (UniqueName: \"kubernetes.io/projected/bc5ede45-712d-49a3-b8ba-e984a8de81c3-kube-api-access-h9jdc\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.275654 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-sysconfig\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.275713 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275667    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c9w6\" (UniqueName: \"kubernetes.io/projected/a13e4959-f5b7-4ad9-a9c1-462cb376231b-kube-api-access-7c9w6\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.275713 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275699    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6mrp\" (UniqueName: \"kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp\") pod \"network-check-target-tlx9j\" (UID: \"db50e6b2-f690-4c96-a2f6-1843a40e95ec\") " pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:25.275810 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275735    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/434a4424-1ea6-4226-8665-6178bcdc10a2-host-slash\") pod \"iptables-alerter-tmnd6\" (UID: \"434a4424-1ea6-4226-8665-6178bcdc10a2\") " pod="openshift-network-operator/iptables-alerter-tmnd6"
Jul 02 13:44:25.275810 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275780    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbg22\" (UniqueName: \"kubernetes.io/projected/434a4424-1ea6-4226-8665-6178bcdc10a2-kube-api-access-rbg22\") pod \"iptables-alerter-tmnd6\" (UID: \"434a4424-1ea6-4226-8665-6178bcdc10a2\") " pod="openshift-network-operator/iptables-alerter-tmnd6"
Jul 02 13:44:25.275810 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275798    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:25.275951 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275820    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac51af3-0564-4b8c-946d-7ce5d5bbdf7e-tmp-dir\") pod \"node-resolver-sdckb\" (UID: \"fac51af3-0564-4b8c-946d-7ce5d5bbdf7e\") " pod="openshift-dns/node-resolver-sdckb"
Jul 02 13:44:25.275951 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275863    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-modprobe-d\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.275951 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275899    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-kubernetes\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.276099 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.275963    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-host\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.276099 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276016    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9chs\" (UniqueName: \"kubernetes.io/projected/fac51af3-0564-4b8c-946d-7ce5d5bbdf7e-kube-api-access-t9chs\") pod \"node-resolver-sdckb\" (UID: \"fac51af3-0564-4b8c-946d-7ce5d5bbdf7e\") " pod="openshift-dns/node-resolver-sdckb"
Jul 02 13:44:25.276099 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276076    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:25.276277 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276143    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.276277 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276178    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c-host\") pod \"node-ca-d4gsb\" (UID: \"fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c\") " pod="openshift-image-registry/node-ca-d4gsb"
Jul 02 13:44:25.276413 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276277    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-sysctl-d\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.276413 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276331    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-sysctl-conf\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.276413 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276392    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-systemd\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.276560 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276443    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-run\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.276560 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276477    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-sys\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.276560 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276505    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-socket-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.276560 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276548    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c-serviceca\") pod \"node-ca-d4gsb\" (UID: \"fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c\") " pod="openshift-image-registry/node-ca-d4gsb"
Jul 02 13:44:25.276749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wswmh\" (UniqueName: \"kubernetes.io/projected/fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c-kube-api-access-wswmh\") pod \"node-ca-d4gsb\" (UID: \"fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c\") " pod="openshift-image-registry/node-ca-d4gsb"
Jul 02 13:44:25.276749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a13e4959-f5b7-4ad9-a9c1-462cb376231b-tmp\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.276749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276626    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 13:44:25.276749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276651    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-device-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.276749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276686    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-var-lib-kubelet\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.276993 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276850    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f2c66\""
Jul 02 13:44:25.276993 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276925    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77c36603-0176-42fd-a5db-790d4cb28509-kubelet-config\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:25.277095 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.276985    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77c36603-0176-42fd-a5db-790d4cb28509-dbus\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:25.277095 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277029    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 13:44:25.277095 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277048    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:25.277272 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277164    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znpph\" (UniqueName: \"kubernetes.io/projected/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-kube-api-access-znpph\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:25.277272 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277205    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 13:44:25.277272 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277215    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/434a4424-1ea6-4226-8665-6178bcdc10a2-iptables-alerter-script\") pod \"iptables-alerter-tmnd6\" (UID: \"434a4424-1ea6-4226-8665-6178bcdc10a2\") " pod="openshift-network-operator/iptables-alerter-tmnd6"
Jul 02 13:44:25.277272 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fac51af3-0564-4b8c-946d-7ce5d5bbdf7e-hosts-file\") pod \"node-resolver-sdckb\" (UID: \"fac51af3-0564-4b8c-946d-7ce5d5bbdf7e\") " pod="openshift-dns/node-resolver-sdckb"
Jul 02 13:44:25.277467 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-lib-modules\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.277467 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-tuned\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.277467 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277344    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 13:44:25.277618 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277540    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 13:44:25.277672 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277623    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 13:44:25.277672 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.277632    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sxmmt\""
Jul 02 13:44:25.280062 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.280040    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 13:44:25.280317 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.280294    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:44:25.280607 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.280588    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 13:44:25.280695 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.280643    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xv2cj\""
Jul 02 13:44:25.303275 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.303256    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v84n5"
Jul 02 13:44:25.316259 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.316239    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v84n5"
Jul 02 13:44:25.349389 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.349337    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-214.ec2.internal" event={"ID":"d47d563e7b550e551dc66f6bd2f8408e","Type":"ContainerStarted","Data":"e6fd68e2413ae75a9424fba7c49fdb68c8744e45576c8f021f7b9e9fe09d4855"}
Jul 02 13:44:25.350481 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.350456    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal" event={"ID":"c08fb0291bc2eef1de979b2d2e6320f7","Type":"ContainerStarted","Data":"5d369e5cc5e92371f40cf6e8542102c15affbceffe451ac4dcb5aa3700b2db63"}
Jul 02 13:44:25.366386 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.366366    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 13:44:25.377484 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-hostroot\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.377603 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377493    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be6b5fc4-8ec4-4bf6-8386-9918c653640a-cni-binary-copy\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.377603 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377515    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:25.377603 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377534    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-system-cni-dir\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.377603 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377563    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-sysctl-d\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.377603 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-sysctl-conf\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.377603 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377595    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-systemd\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-run\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377655    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-systemd\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.377690    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-sys\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377746    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lln2n\" (UniqueName: \"kubernetes.io/projected/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-kube-api-access-lln2n\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377757    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-sysctl-d\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377775    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc78s\" (UniqueName: \"kubernetes.io/projected/be6b5fc4-8ec4-4bf6-8386-9918c653640a-kube-api-access-bc78s\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377765    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-sysctl-conf\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377797    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-sys\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wswmh\" (UniqueName: \"kubernetes.io/projected/fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c-kube-api-access-wswmh\") pod \"node-ca-d4gsb\" (UID: \"fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c\") " pod="openshift-image-registry/node-ca-d4gsb"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377794    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-run\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.377825    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret podName:77c36603-0176-42fd-a5db-790d4cb28509 nodeName:}" failed. No retries permitted until 2026-07-02 13:44:25.877799305 +0000 UTC m=+3.073422796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret") pod "global-pull-secret-syncer-h9rfz" (UID: "77c36603-0176-42fd-a5db-790d4cb28509") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-systemd-units\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.377911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-etc-openvswitch\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-cni-netd\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377953    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77c36603-0176-42fd-a5db-790d4cb28509-kubelet-config\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377978    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77c36603-0176-42fd-a5db-790d4cb28509-dbus\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.377994    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77c36603-0176-42fd-a5db-790d4cb28509-kubelet-config\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378004    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpph\" (UniqueName: \"kubernetes.io/projected/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-kube-api-access-znpph\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/434a4424-1ea6-4226-8665-6178bcdc10a2-iptables-alerter-script\") pod \"iptables-alerter-tmnd6\" (UID: \"434a4424-1ea6-4226-8665-6178bcdc10a2\") " pod="openshift-network-operator/iptables-alerter-tmnd6"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378097    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77c36603-0176-42fd-a5db-790d4cb28509-dbus\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fac51af3-0564-4b8c-946d-7ce5d5bbdf7e-hosts-file\") pod \"node-resolver-sdckb\" (UID: \"fac51af3-0564-4b8c-946d-7ce5d5bbdf7e\") " pod="openshift-dns/node-resolver-sdckb"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.378131    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-log-socket\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378165    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fac51af3-0564-4b8c-946d-7ce5d5bbdf7e-hosts-file\") pod \"node-resolver-sdckb\" (UID: \"fac51af3-0564-4b8c-946d-7ce5d5bbdf7e\") " pod="openshift-dns/node-resolver-sdckb"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.378208    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs podName:fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de nodeName:}" failed. No retries permitted until 2026-07-02 13:44:25.878190365 +0000 UTC m=+3.073813873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs") pod "network-metrics-daemon-6vhv6" (UID: "fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378225    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-tuned\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378286    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-sys-fs\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.378684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378303    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-run-openvswitch\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378319    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-system-cni-dir\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378343    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mrp\" (UniqueName: \"kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp\") pod \"network-check-target-tlx9j\" (UID: \"db50e6b2-f690-4c96-a2f6-1843a40e95ec\") " pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378381    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-socket-dir-parent\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378411    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/11a5eb9e-bf6c-4acc-8566-3047e682e934-agent-certs\") pod \"konnectivity-agent-rthpc\" (UID: \"11a5eb9e-bf6c-4acc-8566-3047e682e934\") " pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378423    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-sys-fs\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378426    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbg22\" (UniqueName: \"kubernetes.io/projected/434a4424-1ea6-4226-8665-6178bcdc10a2-kube-api-access-rbg22\") pod \"iptables-alerter-tmnd6\" (UID: \"434a4424-1ea6-4226-8665-6178bcdc10a2\") " pod="openshift-network-operator/iptables-alerter-tmnd6"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-kubernetes\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-host\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378569    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-var-lib-cni-bin\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378554    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9310de27-67d9-48e4-9628-ac3302f1b57d-ovnkube-config\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378601    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-host\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378681    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9chs\" (UniqueName: \"kubernetes.io/projected/fac51af3-0564-4b8c-946d-7ce5d5bbdf7e-kube-api-access-t9chs\") pod \"node-resolver-sdckb\" (UID: \"fac51af3-0564-4b8c-946d-7ce5d5bbdf7e\") " pod="openshift-dns/node-resolver-sdckb"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378710    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-conf-dir\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.379610 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378735    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-etc-kubernetes\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378762    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-cni-bin\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378791    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/434a4424-1ea6-4226-8665-6178bcdc10a2-iptables-alerter-script\") pod \"iptables-alerter-tmnd6\" (UID: \"434a4424-1ea6-4226-8665-6178bcdc10a2\") " pod="openshift-network-operator/iptables-alerter-tmnd6"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378587    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-kubernetes\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.378951    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-cni-dir\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-cni-binary-copy\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379016    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379050    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9310de27-67d9-48e4-9628-ac3302f1b57d-ovn-node-metrics-cert\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379117    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c-host\") pod \"node-ca-d4gsb\" (UID: \"fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c\") " pod="openshift-image-registry/node-ca-d4gsb"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-socket-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379181    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-run-netns\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379184    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c-host\") pod \"node-ca-d4gsb\" (UID: \"fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c\") " pod="openshift-image-registry/node-ca-d4gsb"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9310de27-67d9-48e4-9628-ac3302f1b57d-ovnkube-script-lib\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379255    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c-serviceca\") pod \"node-ca-d4gsb\" (UID: \"fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c\") " pod="openshift-image-registry/node-ca-d4gsb"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379281    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-socket-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.380463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379284    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a13e4959-f5b7-4ad9-a9c1-462cb376231b-tmp\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379320    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-device-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-run-multus-certs\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379384    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-run-ovn\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379405    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9310de27-67d9-48e4-9628-ac3302f1b57d-env-overrides\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379427    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-registration-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379452    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9jdc\" (UniqueName: \"kubernetes.io/projected/bc5ede45-712d-49a3-b8ba-e984a8de81c3-kube-api-access-h9jdc\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379477    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-cnibin\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379512    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-run-netns\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-var-lib-kubelet\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379579    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-lib-modules\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379600    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-os-release\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379619    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-slash\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-run-systemd\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-var-lib-openvswitch\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379700    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-node-log\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379704    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-device-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.381255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379720    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-run-ovn-kubernetes\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379742    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw6m4\" (UniqueName: \"kubernetes.io/projected/9310de27-67d9-48e4-9628-ac3302f1b57d-kube-api-access-kw6m4\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-sysconfig\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c9w6\" (UniqueName: \"kubernetes.io/projected/a13e4959-f5b7-4ad9-a9c1-462cb376231b-kube-api-access-7c9w6\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379795    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc5ede45-712d-49a3-b8ba-e984a8de81c3-registration-dir\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-var-lib-kubelet\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379832    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-daemon-config\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379867    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-cnibin\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379914    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/434a4424-1ea6-4226-8665-6178bcdc10a2-host-slash\") pod \"iptables-alerter-tmnd6\" (UID: \"434a4424-1ea6-4226-8665-6178bcdc10a2\") " pod="openshift-network-operator/iptables-alerter-tmnd6"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac51af3-0564-4b8c-946d-7ce5d5bbdf7e-tmp-dir\") pod \"node-resolver-sdckb\" (UID: \"fac51af3-0564-4b8c-946d-7ce5d5bbdf7e\") " pod="openshift-dns/node-resolver-sdckb"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.379985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-modprobe-d\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380085    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-var-lib-kubelet\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380132    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-kubelet\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380151    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-sysconfig\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380163    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be6b5fc4-8ec4-4bf6-8386-9918c653640a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380169    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/434a4424-1ea6-4226-8665-6178bcdc10a2-host-slash\") pod \"iptables-alerter-tmnd6\" (UID: \"434a4424-1ea6-4226-8665-6178bcdc10a2\") " pod="openshift-network-operator/iptables-alerter-tmnd6"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380194    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be6b5fc4-8ec4-4bf6-8386-9918c653640a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.381932 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/11a5eb9e-bf6c-4acc-8566-3047e682e934-konnectivity-ca\") pod \"konnectivity-agent-rthpc\" (UID: \"11a5eb9e-bf6c-4acc-8566-3047e682e934\") " pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:25.382489 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380280    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-os-release\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.382489 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-run-k8s-cni-cncf-io\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.382489 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-var-lib-cni-multus\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.382489 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380344    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-lib-modules\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.382489 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380486    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-modprobe-d\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.382489 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380488    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c-serviceca\") pod \"node-ca-d4gsb\" (UID: \"fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c\") " pod="openshift-image-registry/node-ca-d4gsb"
Jul 02 13:44:25.382489 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.380776    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac51af3-0564-4b8c-946d-7ce5d5bbdf7e-tmp-dir\") pod \"node-resolver-sdckb\" (UID: \"fac51af3-0564-4b8c-946d-7ce5d5bbdf7e\") " pod="openshift-dns/node-resolver-sdckb"
Jul 02 13:44:25.382489 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.382439    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a13e4959-f5b7-4ad9-a9c1-462cb376231b-etc-tuned\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.382806 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.382586    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a13e4959-f5b7-4ad9-a9c1-462cb376231b-tmp\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.389311 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.389292    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:44:25.389403 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.389318    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:44:25.389403 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.389328    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p6mrp for pod openshift-network-diagnostics/network-check-target-tlx9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:25.389403 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.389386    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp podName:db50e6b2-f690-4c96-a2f6-1843a40e95ec nodeName:}" failed. No retries permitted until 2026-07-02 13:44:25.889369356 +0000 UTC m=+3.084992866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p6mrp" (UniqueName: "kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp") pod "network-check-target-tlx9j" (UID: "db50e6b2-f690-4c96-a2f6-1843a40e95ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:25.401131 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.401082    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9chs\" (UniqueName: \"kubernetes.io/projected/fac51af3-0564-4b8c-946d-7ce5d5bbdf7e-kube-api-access-t9chs\") pod \"node-resolver-sdckb\" (UID: \"fac51af3-0564-4b8c-946d-7ce5d5bbdf7e\") " pod="openshift-dns/node-resolver-sdckb"
Jul 02 13:44:25.401708 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.401678    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znpph\" (UniqueName: \"kubernetes.io/projected/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-kube-api-access-znpph\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:25.401923 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.401863    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9jdc\" (UniqueName: \"kubernetes.io/projected/bc5ede45-712d-49a3-b8ba-e984a8de81c3-kube-api-access-h9jdc\") pod \"aws-ebs-csi-driver-node-dgkv6\" (UID: \"bc5ede45-712d-49a3-b8ba-e984a8de81c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.402321 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.402293    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbg22\" (UniqueName: \"kubernetes.io/projected/434a4424-1ea6-4226-8665-6178bcdc10a2-kube-api-access-rbg22\") pod \"iptables-alerter-tmnd6\" (UID: \"434a4424-1ea6-4226-8665-6178bcdc10a2\") " pod="openshift-network-operator/iptables-alerter-tmnd6"
Jul 02 13:44:25.402582 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.402559    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c9w6\" (UniqueName: \"kubernetes.io/projected/a13e4959-f5b7-4ad9-a9c1-462cb376231b-kube-api-access-7c9w6\") pod \"tuned-lw6s2\" (UID: \"a13e4959-f5b7-4ad9-a9c1-462cb376231b\") " pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.402885 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.402862    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wswmh\" (UniqueName: \"kubernetes.io/projected/fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c-kube-api-access-wswmh\") pod \"node-ca-d4gsb\" (UID: \"fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c\") " pod="openshift-image-registry/node-ca-d4gsb"
Jul 02 13:44:25.480985 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.480950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-conf-dir\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.480985 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.480986    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-etc-kubernetes\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.481245 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481045    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-conf-dir\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.481245 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481102    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-etc-kubernetes\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.481245 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-cni-bin\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481245 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481245 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-cni-dir\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.481245 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481203    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-cni-binary-copy\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.481245 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481220    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9310de27-67d9-48e4-9628-ac3302f1b57d-ovn-node-metrics-cert\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481245 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481226    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481201    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-cni-bin\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481244    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-run-netns\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481294    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-run-netns\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481307    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9310de27-67d9-48e4-9628-ac3302f1b57d-ovnkube-script-lib\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481320    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-cni-dir\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-run-multus-certs\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-run-ovn\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481398    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9310de27-67d9-48e4-9628-ac3302f1b57d-env-overrides\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481446    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-run-multus-certs\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481453    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-run-ovn\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-cnibin\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-run-netns\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481541    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-os-release\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481566    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-slash\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481590    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-run-systemd\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481615    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-var-lib-openvswitch\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-node-log\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.481670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481674    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-os-release\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481687    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-node-log\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481700    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-run-systemd\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481725    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-var-lib-openvswitch\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481611    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-cnibin\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481738    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-run-netns\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-run-ovn-kubernetes\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481773    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-slash\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481779    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-run-ovn-kubernetes\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw6m4\" (UniqueName: \"kubernetes.io/projected/9310de27-67d9-48e4-9628-ac3302f1b57d-kube-api-access-kw6m4\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-var-lib-kubelet\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-daemon-config\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481855    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-cni-binary-copy\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481876    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-cnibin\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481907    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-kubelet\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481913    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-var-lib-kubelet\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481934    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be6b5fc4-8ec4-4bf6-8386-9918c653640a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.482606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481962    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-cnibin\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.481962    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be6b5fc4-8ec4-4bf6-8386-9918c653640a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/11a5eb9e-bf6c-4acc-8566-3047e682e934-konnectivity-ca\") pod \"konnectivity-agent-rthpc\" (UID: \"11a5eb9e-bf6c-4acc-8566-3047e682e934\") " pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482061    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9310de27-67d9-48e4-9628-ac3302f1b57d-ovnkube-script-lib\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482137    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-kubelet\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482216    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9310de27-67d9-48e4-9628-ac3302f1b57d-env-overrides\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482261    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-os-release\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-run-k8s-cni-cncf-io\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-var-lib-cni-multus\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482338    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-hostroot\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482354    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-os-release\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482364    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be6b5fc4-8ec4-4bf6-8386-9918c653640a-cni-binary-copy\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482413    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-system-cni-dir\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482417    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-run-k8s-cni-cncf-io\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482444    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lln2n\" (UniqueName: \"kubernetes.io/projected/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-kube-api-access-lln2n\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc78s\" (UniqueName: \"kubernetes.io/projected/be6b5fc4-8ec4-4bf6-8386-9918c653640a-kube-api-access-bc78s\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482507    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-systemd-units\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.483470 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482533    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-etc-openvswitch\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-cni-netd\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-log-socket\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482622    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/11a5eb9e-bf6c-4acc-8566-3047e682e934-konnectivity-ca\") pod \"konnectivity-agent-rthpc\" (UID: \"11a5eb9e-bf6c-4acc-8566-3047e682e934\") " pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482635    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-daemon-config\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482637    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-run-openvswitch\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482678    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-var-lib-cni-multus\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482689    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-system-cni-dir\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482704    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-etc-openvswitch\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482718    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-run-openvswitch\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-socket-dir-parent\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482756    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-systemd-units\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482756    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/11a5eb9e-bf6c-4acc-8566-3047e682e934-agent-certs\") pod \"konnectivity-agent-rthpc\" (UID: \"11a5eb9e-bf6c-4acc-8566-3047e682e934\") " pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482773    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-system-cni-dir\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482798    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-system-cni-dir\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482807    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-host-cni-netd\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482817    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9310de27-67d9-48e4-9628-ac3302f1b57d-log-socket\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.484168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482654    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be6b5fc4-8ec4-4bf6-8386-9918c653640a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.484749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482801    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-var-lib-cni-bin\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.484749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482855    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-multus-socket-dir-parent\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.484749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482870    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-host-var-lib-cni-bin\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.484749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482873    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-hostroot\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.484749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482888    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9310de27-67d9-48e4-9628-ac3302f1b57d-ovnkube-config\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.484749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.482917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.484749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.483087    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be6b5fc4-8ec4-4bf6-8386-9918c653640a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.484749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.483392    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9310de27-67d9-48e4-9628-ac3302f1b57d-ovnkube-config\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.484749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.483575    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be6b5fc4-8ec4-4bf6-8386-9918c653640a-cni-binary-copy\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.484749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.483579    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be6b5fc4-8ec4-4bf6-8386-9918c653640a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.484749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.484061    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9310de27-67d9-48e4-9628-ac3302f1b57d-ovn-node-metrics-cert\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.485305 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.485289    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/11a5eb9e-bf6c-4acc-8566-3047e682e934-agent-certs\") pod \"konnectivity-agent-rthpc\" (UID: \"11a5eb9e-bf6c-4acc-8566-3047e682e934\") " pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:25.495222 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.495187    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw6m4\" (UniqueName: \"kubernetes.io/projected/9310de27-67d9-48e4-9628-ac3302f1b57d-kube-api-access-kw6m4\") pod \"ovnkube-node-lmvr2\" (UID: \"9310de27-67d9-48e4-9628-ac3302f1b57d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.496109 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.496078    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc78s\" (UniqueName: \"kubernetes.io/projected/be6b5fc4-8ec4-4bf6-8386-9918c653640a-kube-api-access-bc78s\") pod \"multus-additional-cni-plugins-wvh8f\" (UID: \"be6b5fc4-8ec4-4bf6-8386-9918c653640a\") " pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.496477 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.496460    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lln2n\" (UniqueName: \"kubernetes.io/projected/8319cd50-4436-4a2c-9a5d-0c8c740c0fbc-kube-api-access-lln2n\") pod \"multus-nqrn8\" (UID: \"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc\") " pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.574666 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.574630    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d4gsb"
Jul 02 13:44:25.580077 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.580051    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tmnd6"
Jul 02 13:44:25.592293 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.592271    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sdckb"
Jul 02 13:44:25.596390 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.596369    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lw6s2"
Jul 02 13:44:25.605442 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.605419    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6"
Jul 02 13:44:25.614331 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.614308    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nqrn8"
Jul 02 13:44:25.622275 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.622253    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:25.630223 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.630191    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wvh8f"
Jul 02 13:44:25.637181 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.637162    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:25.887944 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.887844    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:25.887944 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.887931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:25.888172 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.887990    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:25.888172 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.888063    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret podName:77c36603-0176-42fd-a5db-790d4cb28509 nodeName:}" failed. No retries permitted until 2026-07-02 13:44:26.888043404 +0000 UTC m=+4.083666893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret") pod "global-pull-secret-syncer-h9rfz" (UID: "77c36603-0176-42fd-a5db-790d4cb28509") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:25.888172 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.888075    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:25.888172 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.888133    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs podName:fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de nodeName:}" failed. No retries permitted until 2026-07-02 13:44:26.888115786 +0000 UTC m=+4.083739289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs") pod "network-metrics-daemon-6vhv6" (UID: "fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:25.988350 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:25.988303    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mrp\" (UniqueName: \"kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp\") pod \"network-check-target-tlx9j\" (UID: \"db50e6b2-f690-4c96-a2f6-1843a40e95ec\") " pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:25.988548 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.988506    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:44:25.988548 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.988538    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:44:25.988641 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.988553    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p6mrp for pod openshift-network-diagnostics/network-check-target-tlx9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:25.988641 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:25.988622    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp podName:db50e6b2-f690-4c96-a2f6-1843a40e95ec nodeName:}" failed. No retries permitted until 2026-07-02 13:44:26.988601562 +0000 UTC m=+4.184225054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6mrp" (UniqueName: "kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp") pod "network-check-target-tlx9j" (UID: "db50e6b2-f690-4c96-a2f6-1843a40e95ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:26.288033 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:26.287991    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:44:26.320550 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:26.320508    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:39:25 +0000 UTC" deadline="2028-02-06 23:48:35.695802741 +0000 UTC"
Jul 02 13:44:26.320550 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:26.320542    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14026h4m9.375265783s"
Jul 02 13:44:26.481309 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:26.481277    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434a4424_1ea6_4226_8665_6178bcdc10a2.slice/crio-12592b7b699bb94d3ccf4850da964e87207ee30016090236029119c9697c5a12 WatchSource:0}: Error finding container 12592b7b699bb94d3ccf4850da964e87207ee30016090236029119c9697c5a12: Status 404 returned error can't find the container with id 12592b7b699bb94d3ccf4850da964e87207ee30016090236029119c9697c5a12
Jul 02 13:44:26.483851 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:26.483788    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd1955d_c9c4_4724_ab2c_6d3fec6d9b5c.slice/crio-d9ab268e2ae78822be5370952f7a39b227c5d033ef43a685d959632d038b32ff WatchSource:0}: Error finding container d9ab268e2ae78822be5370952f7a39b227c5d033ef43a685d959632d038b32ff: Status 404 returned error can't find the container with id d9ab268e2ae78822be5370952f7a39b227c5d033ef43a685d959632d038b32ff
Jul 02 13:44:26.488845 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:26.488720    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8319cd50_4436_4a2c_9a5d_0c8c740c0fbc.slice/crio-63fecbe002010c102ba6f41b126cf47bff9834799efca4a2b08717a29a537e76 WatchSource:0}: Error finding container 63fecbe002010c102ba6f41b126cf47bff9834799efca4a2b08717a29a537e76: Status 404 returned error can't find the container with id 63fecbe002010c102ba6f41b126cf47bff9834799efca4a2b08717a29a537e76
Jul 02 13:44:26.490171 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:26.490143    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9310de27_67d9_48e4_9628_ac3302f1b57d.slice/crio-c98f9e17bfee67b7129eea4230766645dbd0bed9e78d9e6f391641484b93520d WatchSource:0}: Error finding container c98f9e17bfee67b7129eea4230766645dbd0bed9e78d9e6f391641484b93520d: Status 404 returned error can't find the container with id c98f9e17bfee67b7129eea4230766645dbd0bed9e78d9e6f391641484b93520d
Jul 02 13:44:26.491302 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:26.491277    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13e4959_f5b7_4ad9_a9c1_462cb376231b.slice/crio-7263369e267155642ccee05f8b0b6f2dded14e0b771174502ad13712bd03e86c WatchSource:0}: Error finding container 7263369e267155642ccee05f8b0b6f2dded14e0b771174502ad13712bd03e86c: Status 404 returned error can't find the container with id 7263369e267155642ccee05f8b0b6f2dded14e0b771174502ad13712bd03e86c
Jul 02 13:44:26.492989 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:26.492865    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a5eb9e_bf6c_4acc_8566_3047e682e934.slice/crio-9361e70e21f6d29ac6ddd43a5f4887db851a6d3ceac90c7e0b6a509a7573f86e WatchSource:0}: Error finding container 9361e70e21f6d29ac6ddd43a5f4887db851a6d3ceac90c7e0b6a509a7573f86e: Status 404 returned error can't find the container with id 9361e70e21f6d29ac6ddd43a5f4887db851a6d3ceac90c7e0b6a509a7573f86e
Jul 02 13:44:26.494111 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:26.493925    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5ede45_712d_49a3_b8ba_e984a8de81c3.slice/crio-48acb95aee342430b1773f59eb948e084df62173381bf48dca66797fb7b3540c WatchSource:0}: Error finding container 48acb95aee342430b1773f59eb948e084df62173381bf48dca66797fb7b3540c: Status 404 returned error can't find the container with id 48acb95aee342430b1773f59eb948e084df62173381bf48dca66797fb7b3540c
Jul 02 13:44:26.495230 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:26.494878    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6b5fc4_8ec4_4bf6_8386_9918c653640a.slice/crio-d50268467e8ef39418bfe8066035645c5b54fa13231dff9588ce580b0180502f WatchSource:0}: Error finding container d50268467e8ef39418bfe8066035645c5b54fa13231dff9588ce580b0180502f: Status 404 returned error can't find the container with id d50268467e8ef39418bfe8066035645c5b54fa13231dff9588ce580b0180502f
Jul 02 13:44:26.894891 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:26.894702    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:26.895039 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:26.894903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:26.895039 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:26.894849    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:26.895039 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:26.895014    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret podName:77c36603-0176-42fd-a5db-790d4cb28509 nodeName:}" failed. No retries permitted until 2026-07-02 13:44:28.894997735 +0000 UTC m=+6.090621222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret") pod "global-pull-secret-syncer-h9rfz" (UID: "77c36603-0176-42fd-a5db-790d4cb28509") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:26.895039 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:26.895025    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:26.895220 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:26.895065    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs podName:fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de nodeName:}" failed. No retries permitted until 2026-07-02 13:44:28.895052629 +0000 UTC m=+6.090676116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs") pod "network-metrics-daemon-6vhv6" (UID: "fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:26.996236 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:26.996004    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mrp\" (UniqueName: \"kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp\") pod \"network-check-target-tlx9j\" (UID: \"db50e6b2-f690-4c96-a2f6-1843a40e95ec\") " pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:26.996236 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:26.996194    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:44:26.996236 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:26.996218    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:44:26.996236 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:26.996232    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p6mrp for pod openshift-network-diagnostics/network-check-target-tlx9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:26.996579 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:26.996300    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp podName:db50e6b2-f690-4c96-a2f6-1843a40e95ec nodeName:}" failed. No retries permitted until 2026-07-02 13:44:28.996281089 +0000 UTC m=+6.191904579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6mrp" (UniqueName: "kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp") pod "network-check-target-tlx9j" (UID: "db50e6b2-f690-4c96-a2f6-1843a40e95ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:27.320794 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.320671    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:39:25 +0000 UTC" deadline="2027-11-28 22:18:59.749946494 +0000 UTC"
Jul 02 13:44:27.320794 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.320791    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12344h34m32.429159518s"
Jul 02 13:44:27.348266 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.348237    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:27.348423 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:27.348371    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:27.348859 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.348835    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:27.348967 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:27.348947    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:27.349066 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.349050    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:27.349173 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:27.349148    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:27.366463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.366397    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lw6s2" event={"ID":"a13e4959-f5b7-4ad9-a9c1-462cb376231b","Type":"ContainerStarted","Data":"7263369e267155642ccee05f8b0b6f2dded14e0b771174502ad13712bd03e86c"}
Jul 02 13:44:27.380240 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.380180    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nqrn8" event={"ID":"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc","Type":"ContainerStarted","Data":"63fecbe002010c102ba6f41b126cf47bff9834799efca4a2b08717a29a537e76"}
Jul 02 13:44:27.386773 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.386744    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d4gsb" event={"ID":"fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c","Type":"ContainerStarted","Data":"d9ab268e2ae78822be5370952f7a39b227c5d033ef43a685d959632d038b32ff"}
Jul 02 13:44:27.390452 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.390421    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tmnd6" event={"ID":"434a4424-1ea6-4226-8665-6178bcdc10a2","Type":"ContainerStarted","Data":"12592b7b699bb94d3ccf4850da964e87207ee30016090236029119c9697c5a12"}
Jul 02 13:44:27.409797 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.409501    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-214.ec2.internal" event={"ID":"d47d563e7b550e551dc66f6bd2f8408e","Type":"ContainerStarted","Data":"baa7cab589aefc0beb975554a54085e2ceefb1154b10fd01569aade99ad2e265"}
Jul 02 13:44:27.417193 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.417121    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rthpc" event={"ID":"11a5eb9e-bf6c-4acc-8566-3047e682e934","Type":"ContainerStarted","Data":"9361e70e21f6d29ac6ddd43a5f4887db851a6d3ceac90c7e0b6a509a7573f86e"}
Jul 02 13:44:27.419305 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.419168    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2" event={"ID":"9310de27-67d9-48e4-9628-ac3302f1b57d","Type":"ContainerStarted","Data":"c98f9e17bfee67b7129eea4230766645dbd0bed9e78d9e6f391641484b93520d"}
Jul 02 13:44:27.436639 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.435907    2574 generic.go:356] "Generic (PLEG): container finished" podID="c08fb0291bc2eef1de979b2d2e6320f7" containerID="a43084cc4e2f172859632ed5c33155c2683f3fd23c7c13492bb9b6ad2f58050e" exitCode=0
Jul 02 13:44:27.436639 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.435983    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal" event={"ID":"c08fb0291bc2eef1de979b2d2e6320f7","Type":"ContainerDied","Data":"a43084cc4e2f172859632ed5c33155c2683f3fd23c7c13492bb9b6ad2f58050e"}
Jul 02 13:44:27.457497 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.457442    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sdckb" event={"ID":"fac51af3-0564-4b8c-946d-7ce5d5bbdf7e","Type":"ContainerStarted","Data":"470d42b6d9ea4468e5fb86680f566553c929ab783a1448da9e74dec535577271"}
Jul 02 13:44:27.472443 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.472117    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-214.ec2.internal" podStartSLOduration=3.472098581 podStartE2EDuration="3.472098581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:44:27.447204723 +0000 UTC m=+4.642828242" watchObservedRunningTime="2026-07-02 13:44:27.472098581 +0000 UTC m=+4.667722090"
Jul 02 13:44:27.479478 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.479203    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvh8f" event={"ID":"be6b5fc4-8ec4-4bf6-8386-9918c653640a","Type":"ContainerStarted","Data":"d50268467e8ef39418bfe8066035645c5b54fa13231dff9588ce580b0180502f"}
Jul 02 13:44:27.482531 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:27.482468    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6" event={"ID":"bc5ede45-712d-49a3-b8ba-e984a8de81c3","Type":"ContainerStarted","Data":"48acb95aee342430b1773f59eb948e084df62173381bf48dca66797fb7b3540c"}
Jul 02 13:44:28.523889 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:28.523490    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal" event={"ID":"c08fb0291bc2eef1de979b2d2e6320f7","Type":"ContainerStarted","Data":"582c2f35611ce643ec5d255e29b12d81890047a8df17bb7a77d39673d5718654"}
Jul 02 13:44:28.911796 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:28.911762    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:28.911982 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:28.911819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:28.911982 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:28.911963    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:28.912092 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:28.912023    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs podName:fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de nodeName:}" failed. No retries permitted until 2026-07-02 13:44:32.912004224 +0000 UTC m=+10.107627712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs") pod "network-metrics-daemon-6vhv6" (UID: "fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:28.912436 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:28.912419    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:28.912529 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:28.912470    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret podName:77c36603-0176-42fd-a5db-790d4cb28509 nodeName:}" failed. No retries permitted until 2026-07-02 13:44:32.912454731 +0000 UTC m=+10.108078216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret") pod "global-pull-secret-syncer-h9rfz" (UID: "77c36603-0176-42fd-a5db-790d4cb28509") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:29.014098 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:29.014057    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mrp\" (UniqueName: \"kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp\") pod \"network-check-target-tlx9j\" (UID: \"db50e6b2-f690-4c96-a2f6-1843a40e95ec\") " pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:29.014287 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:29.014259    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:44:29.014287 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:29.014277    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:44:29.014410 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:29.014291    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p6mrp for pod openshift-network-diagnostics/network-check-target-tlx9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:29.014410 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:29.014369    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp podName:db50e6b2-f690-4c96-a2f6-1843a40e95ec nodeName:}" failed. No retries permitted until 2026-07-02 13:44:33.014351636 +0000 UTC m=+10.209975123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6mrp" (UniqueName: "kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp") pod "network-check-target-tlx9j" (UID: "db50e6b2-f690-4c96-a2f6-1843a40e95ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:29.346320 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:29.346242    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:29.346484 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:29.346373    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:29.346774 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:29.346751    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:29.346897 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:29.346859    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:29.346993 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:29.346951    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:29.347044 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:29.347020    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:31.345326 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:31.345240    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:31.345759 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:31.345364    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:31.345964 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:31.345809    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:31.345964 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:31.345913    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:31.348170 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:31.346213    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:31.348170 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:31.346314    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:32.946198 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:32.946152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:32.946745 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:32.946211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:32.946745 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:32.946338    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:32.946745 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:32.946369    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:32.946745 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:32.946397    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs podName:fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de nodeName:}" failed. No retries permitted until 2026-07-02 13:44:40.946379167 +0000 UTC m=+18.142002655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs") pod "network-metrics-daemon-6vhv6" (UID: "fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:32.946745 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:32.946450    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret podName:77c36603-0176-42fd-a5db-790d4cb28509 nodeName:}" failed. No retries permitted until 2026-07-02 13:44:40.946431013 +0000 UTC m=+18.142054502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret") pod "global-pull-secret-syncer-h9rfz" (UID: "77c36603-0176-42fd-a5db-790d4cb28509") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:33.047535 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:33.047498    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mrp\" (UniqueName: \"kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp\") pod \"network-check-target-tlx9j\" (UID: \"db50e6b2-f690-4c96-a2f6-1843a40e95ec\") " pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:33.047770 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:33.047740    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:44:33.047770 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:33.047769    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:44:33.047972 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:33.047784    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p6mrp for pod openshift-network-diagnostics/network-check-target-tlx9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:33.047972 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:33.047847    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp podName:db50e6b2-f690-4c96-a2f6-1843a40e95ec nodeName:}" failed. No retries permitted until 2026-07-02 13:44:41.047827916 +0000 UTC m=+18.243451415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6mrp" (UniqueName: "kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp") pod "network-check-target-tlx9j" (UID: "db50e6b2-f690-4c96-a2f6-1843a40e95ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:33.352150 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:33.352060    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:33.352295 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:33.352183    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:33.352295 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:33.352259    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:33.352422 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:33.352394    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:33.352474 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:33.352453    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:33.352557 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:33.352523    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:35.346318 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:35.346218    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:35.346318 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:35.346274    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:35.346318 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:35.346218    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:35.346883 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:35.346370    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:35.346883 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:35.346466    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:35.346883 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:35.346535    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:37.345816 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:37.345744    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:37.346275 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:37.345742    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:37.346275 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:37.345871    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:37.346275 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:37.346003    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:37.346275 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:37.346033    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:37.346275 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:37.346149    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:39.345313 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:39.345259    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:39.345313 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:39.345302    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:39.345823 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:39.345262    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:39.345823 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:39.345404    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:39.345823 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:39.345486    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:39.345823 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:39.345566    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:41.008071 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:41.008036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:41.008071 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:41.008081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:41.008670 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:41.008209    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:41.008670 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:41.008254    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:41.008670 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:41.008271    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret podName:77c36603-0176-42fd-a5db-790d4cb28509 nodeName:}" failed. No retries permitted until 2026-07-02 13:44:57.008255243 +0000 UTC m=+34.203878728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret") pod "global-pull-secret-syncer-h9rfz" (UID: "77c36603-0176-42fd-a5db-790d4cb28509") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:41.008670 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:41.008306    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs podName:fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de nodeName:}" failed. No retries permitted until 2026-07-02 13:44:57.008290885 +0000 UTC m=+34.203914374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs") pod "network-metrics-daemon-6vhv6" (UID: "fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:41.108977 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:41.108922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mrp\" (UniqueName: \"kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp\") pod \"network-check-target-tlx9j\" (UID: \"db50e6b2-f690-4c96-a2f6-1843a40e95ec\") " pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:41.109180 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:41.109076    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:44:41.109180 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:41.109095    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:44:41.109180 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:41.109110    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p6mrp for pod openshift-network-diagnostics/network-check-target-tlx9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:41.109348 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:41.109187    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp podName:db50e6b2-f690-4c96-a2f6-1843a40e95ec nodeName:}" failed. No retries permitted until 2026-07-02 13:44:57.109168032 +0000 UTC m=+34.304791534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6mrp" (UniqueName: "kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp") pod "network-check-target-tlx9j" (UID: "db50e6b2-f690-4c96-a2f6-1843a40e95ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:41.345240 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:41.345169    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:41.345394 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:41.345164    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:41.345394 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:41.345305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:41.345394 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:41.345302    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:41.345545 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:41.345411    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:41.345545 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:41.345494    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:43.346225 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:43.346190    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:43.346629 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:43.346285    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:43.346629 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:43.346284    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:43.346629 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:43.346402    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:43.346629 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:43.346407    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:43.346629 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:43.346535    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:45.345398 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.345374    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:45.345881 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:45.345474    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:45.345881 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.345508    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:45.345881 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:45.345625    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:45.345881 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.345374    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:45.345881 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:45.345701    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:45.559005 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.558818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lw6s2" event={"ID":"a13e4959-f5b7-4ad9-a9c1-462cb376231b","Type":"ContainerStarted","Data":"6f45431e609d1b9cde5061a5f01e875d195e2e25349db24bac6fa1ab595c927b"}
Jul 02 13:44:45.560119 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.560094    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nqrn8" event={"ID":"8319cd50-4436-4a2c-9a5d-0c8c740c0fbc","Type":"ContainerStarted","Data":"2ebd52a24deeebf8f139b32be6ec24625079828586c3818b92606a9958a4a517"}
Jul 02 13:44:45.561403 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.561377    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d4gsb" event={"ID":"fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c","Type":"ContainerStarted","Data":"7db19523757cf20bc795da8640d421a11cc0f17cf5d0d4af2c574e27a8225027"}
Jul 02 13:44:45.562653 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.562624    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rthpc" event={"ID":"11a5eb9e-bf6c-4acc-8566-3047e682e934","Type":"ContainerStarted","Data":"2851650d4bc2c66ab5a8c690378b7ec22fc77abb405c0375da6ce24c41bde5fd"}
Jul 02 13:44:45.564260 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.564244    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 13:44:45.564516 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.564498    2574 generic.go:356] "Generic (PLEG): container finished" podID="9310de27-67d9-48e4-9628-ac3302f1b57d" containerID="3605227d1be3aa8b32e8ae43a0fe7d0882f3998b8757eb1c07a65b4670a5465d" exitCode=1
Jul 02 13:44:45.564568 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.564556    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2" event={"ID":"9310de27-67d9-48e4-9628-ac3302f1b57d","Type":"ContainerStarted","Data":"0ab4eeece46be37a09c5e5f1f985a31107e7bf0fb04fca00e5c90886de9c2a12"}
Jul 02 13:44:45.564635 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.564620    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2" event={"ID":"9310de27-67d9-48e4-9628-ac3302f1b57d","Type":"ContainerDied","Data":"3605227d1be3aa8b32e8ae43a0fe7d0882f3998b8757eb1c07a65b4670a5465d"}
Jul 02 13:44:45.564682 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.564674    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2" event={"ID":"9310de27-67d9-48e4-9628-ac3302f1b57d","Type":"ContainerStarted","Data":"d09bc102532fa2660fb9058e856213280ba63c9c39ab724f176a3ab834ca39ff"}
Jul 02 13:44:45.565872 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.565853    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sdckb" event={"ID":"fac51af3-0564-4b8c-946d-7ce5d5bbdf7e","Type":"ContainerStarted","Data":"c4dd9cea5c21eec9384139304cf498fb1b9e13534e33a8f8432d88ff1c7e030a"}
Jul 02 13:44:45.567099 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.567082    2574 generic.go:356] "Generic (PLEG): container finished" podID="be6b5fc4-8ec4-4bf6-8386-9918c653640a" containerID="cd83a58a35052f92320ceb437bff786ad149529b045d9c653adfe0bffffec066" exitCode=0
Jul 02 13:44:45.567191 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.567167    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvh8f" event={"ID":"be6b5fc4-8ec4-4bf6-8386-9918c653640a","Type":"ContainerDied","Data":"cd83a58a35052f92320ceb437bff786ad149529b045d9c653adfe0bffffec066"}
Jul 02 13:44:45.568459 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.568440    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6" event={"ID":"bc5ede45-712d-49a3-b8ba-e984a8de81c3","Type":"ContainerStarted","Data":"3c3d9a2321ee2559f17593a9bd698dd89e297157c976be1ab3bf5e9f19155100"}
Jul 02 13:44:45.578005 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.577967    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lw6s2" podStartSLOduration=4.036387727 podStartE2EDuration="22.577955182s" totalImagesPullingTime="18.541567455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:44:45.577507992 +0000 UTC m=+22.773131500" watchObservedRunningTime="2026-07-02 13:44:45.577955182 +0000 UTC m=+22.773578688"
Jul 02 13:44:45.578116 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.578091    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-214.ec2.internal" podStartSLOduration=21.578082693 podStartE2EDuration="21.578082693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:44:28.540809397 +0000 UTC m=+5.736432906" watchObservedRunningTime="2026-07-02 13:44:45.578082693 +0000 UTC m=+22.773706209"
Jul 02 13:44:45.619377 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.619335    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rthpc" podStartSLOduration=12.14741785 podStartE2EDuration="22.619321097s" totalImagesPullingTime="10.471903247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:44:45.618989579 +0000 UTC m=+22.814613090" watchObservedRunningTime="2026-07-02 13:44:45.619321097 +0000 UTC m=+22.814944604"
Jul 02 13:44:45.633595 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.633556    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d4gsb" podStartSLOduration=4.105093636 podStartE2EDuration="22.633541421s" totalImagesPullingTime="18.528447785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:44:45.633322478 +0000 UTC m=+22.828945986" watchObservedRunningTime="2026-07-02 13:44:45.633541421 +0000 UTC m=+22.829164927"
Jul 02 13:44:45.652409 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.652364    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nqrn8" podStartSLOduration=4.075174612 podStartE2EDuration="22.652348842s" totalImagesPullingTime="18.57717423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:44:45.652033326 +0000 UTC m=+22.847656833" watchObservedRunningTime="2026-07-02 13:44:45.652348842 +0000 UTC m=+22.847972349"
Jul 02 13:44:45.669377 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:45.669335    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sdckb" podStartSLOduration=4.152311433 podStartE2EDuration="22.669319472s" totalImagesPullingTime="18.517008039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:44:45.668906071 +0000 UTC m=+22.864529591" watchObservedRunningTime="2026-07-02 13:44:45.669319472 +0000 UTC m=+22.864942978"
Jul 02 13:44:46.508099 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:46.508077    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 13:44:46.572835 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:46.572788    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6" event={"ID":"bc5ede45-712d-49a3-b8ba-e984a8de81c3","Type":"ContainerStarted","Data":"6eb46365259b05e570d71ae0a3c97903365a1a94bba5db046c57f9f74775d037"}
Jul 02 13:44:46.574326 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:46.574297    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tmnd6" event={"ID":"434a4424-1ea6-4226-8665-6178bcdc10a2","Type":"ContainerStarted","Data":"0b097ebbea9f301c767b678a8341dadd1e25dc299bc7362555ba98752557e0b4"}
Jul 02 13:44:46.577198 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:46.577174    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 13:44:46.577588 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:46.577517    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2" event={"ID":"9310de27-67d9-48e4-9628-ac3302f1b57d","Type":"ContainerStarted","Data":"abdef6d0e2714aa4482248a2f5d2cb32b8d5441ac6303f2efa0e0b328eb9e173"}
Jul 02 13:44:46.577682 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:46.577608    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2" event={"ID":"9310de27-67d9-48e4-9628-ac3302f1b57d","Type":"ContainerStarted","Data":"f9627cf2bda6110d71e9da6d2d28f8ba20ca71aca76d6186d8a63a5339e4770d"}
Jul 02 13:44:46.577682 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:46.577661    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2" event={"ID":"9310de27-67d9-48e4-9628-ac3302f1b57d","Type":"ContainerStarted","Data":"7f330ed0ec1dd41de89a1c89a0634cdee63efbf267eeb913200bd64f1e4d951c"}
Jul 02 13:44:46.591285 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:46.591246    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tmnd6" podStartSLOduration=5.057944237 podStartE2EDuration="23.591232029s" totalImagesPullingTime="18.533287792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:44:46.59087495 +0000 UTC m=+23.786498531" watchObservedRunningTime="2026-07-02 13:44:46.591232029 +0000 UTC m=+23.786855535"
Jul 02 13:44:47.345779 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:47.345591    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T13:44:46.50809426Z","UUID":"932d5e9f-68ba-42e9-bd0b-c6d184fde3a4","Handler":null,"Name":"","Endpoint":""}
Jul 02 13:44:47.345779 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:47.345741    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:47.346130 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:47.345741    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:47.346130 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:47.345887    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:47.346130 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:47.345881    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:47.346130 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:47.345966    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:47.346130 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:47.346028    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:47.347484 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:47.347465    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 13:44:47.347593 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:47.347509    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 13:44:47.351618 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:47.350906    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:47.351618 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:47.351049    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:48.584078 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:48.584041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6" event={"ID":"bc5ede45-712d-49a3-b8ba-e984a8de81c3","Type":"ContainerStarted","Data":"00772068c712b3ccf24f18d5b3ea1f5d4e90b3986bfafe349797f92bea7dcedc"}
Jul 02 13:44:48.587539 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:48.587518    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 13:44:48.587949 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:48.587919    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2" event={"ID":"9310de27-67d9-48e4-9628-ac3302f1b57d","Type":"ContainerStarted","Data":"2eed7c677ce97e7f031e46af694e7cb5398b2232ee09adca823714da99164a9b"}
Jul 02 13:44:48.587949 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:48.587946    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 02 13:44:48.603318 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:48.603272    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dgkv6" podStartSLOduration=4.768163245 podStartE2EDuration="25.603255357s" totalImagesPullingTime="20.835092112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:44:48.602863672 +0000 UTC m=+25.798487179" watchObservedRunningTime="2026-07-02 13:44:48.603255357 +0000 UTC m=+25.798878866"
Jul 02 13:44:49.344749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:49.344715    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:49.344749 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:49.344733    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:49.345013 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:49.344728    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:49.345013 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:49.344845    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:49.345013 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:49.344920    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:49.345013 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:49.345002    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:50.596293 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:50.596081    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 13:44:50.597078 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:50.596597    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2" event={"ID":"9310de27-67d9-48e4-9628-ac3302f1b57d","Type":"ContainerStarted","Data":"5644a2abf585470c7e754f0f9da48f44501e4f1823434fcdc992b675b3f91c10"}
Jul 02 13:44:50.597078 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:50.596861    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:50.597078 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:50.596944    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:50.597227 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:50.597085    2574 scope.go:117] "RemoveContainer" containerID="3605227d1be3aa8b32e8ae43a0fe7d0882f3998b8757eb1c07a65b4670a5465d"
Jul 02 13:44:50.598825 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:50.598803    2574 generic.go:356] "Generic (PLEG): container finished" podID="be6b5fc4-8ec4-4bf6-8386-9918c653640a" containerID="4e353b770da68609cfbb9158fe10bacb4a66260594403372c0ff828389e055ee" exitCode=0
Jul 02 13:44:50.598939 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:50.598844    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvh8f" event={"ID":"be6b5fc4-8ec4-4bf6-8386-9918c653640a","Type":"ContainerDied","Data":"4e353b770da68609cfbb9158fe10bacb4a66260594403372c0ff828389e055ee"}
Jul 02 13:44:50.613269 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:50.613247    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:51.345977 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:51.345898    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:51.346187 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:51.345898    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:51.346187 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:51.346034    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:51.346187 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:51.346034    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:51.346355 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:51.346196    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:51.346355 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:51.346240    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:51.604678 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:51.604615    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 13:44:51.605124 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:51.604941    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2" event={"ID":"9310de27-67d9-48e4-9628-ac3302f1b57d","Type":"ContainerStarted","Data":"e446f50e340fa3a189cb0dcfe2746ec9430b6e172a8e0f795319d64812f1f26a"}
Jul 02 13:44:51.605273 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:51.605250    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:51.620416 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:51.620391    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:44:51.637716 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:51.637676    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2" podStartSLOduration=10.038501408 podStartE2EDuration="28.637663224s" totalImagesPullingTime="18.599161816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:44:51.636106003 +0000 UTC m=+28.831729509" watchObservedRunningTime="2026-07-02 13:44:51.637663224 +0000 UTC m=+28.833286731"
Jul 02 13:44:52.017550 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:52.017516    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6vhv6"]
Jul 02 13:44:52.017702 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:52.017657    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:52.017773 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:52.017756    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:52.018084 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:52.018063    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h9rfz"]
Jul 02 13:44:52.018180 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:52.018170    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:52.018256 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:52.018241    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:52.019886 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:52.019860    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tlx9j"]
Jul 02 13:44:52.020002 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:52.019945    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:52.020052 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:52.020013    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:52.609393 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:52.609362    2574 generic.go:356] "Generic (PLEG): container finished" podID="be6b5fc4-8ec4-4bf6-8386-9918c653640a" containerID="dd6310d8b66ef1b082dd84b64b5b8fc3fdcc7a48e0dc98248d8dec3cdd8caa53" exitCode=0
Jul 02 13:44:52.610013 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:52.609422    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvh8f" event={"ID":"be6b5fc4-8ec4-4bf6-8386-9918c653640a","Type":"ContainerDied","Data":"dd6310d8b66ef1b082dd84b64b5b8fc3fdcc7a48e0dc98248d8dec3cdd8caa53"}
Jul 02 13:44:53.345624 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:53.345592    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:53.345802 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:53.345687    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:53.345802 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:53.345763    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:53.345921 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:53.345900    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:54.345552 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:54.345523    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:54.345925 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:54.345616    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:54.442810 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:54.442782    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:54.442954 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:54.442902    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 02 13:44:54.443360 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:54.443340    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rthpc"
Jul 02 13:44:54.616040 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:54.616005    2574 generic.go:356] "Generic (PLEG): container finished" podID="be6b5fc4-8ec4-4bf6-8386-9918c653640a" containerID="f67e1e0be890e8a060da0f136f64fa6eaf25b287fdf76ded2283181a90a30ae0" exitCode=0
Jul 02 13:44:54.616197 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:54.616086    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvh8f" event={"ID":"be6b5fc4-8ec4-4bf6-8386-9918c653640a","Type":"ContainerDied","Data":"f67e1e0be890e8a060da0f136f64fa6eaf25b287fdf76ded2283181a90a30ae0"}
Jul 02 13:44:55.345124 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:55.344949    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:55.345302 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:55.344961    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:55.345302 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:55.345236    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:55.345423 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:55.345319    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:56.344874 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:56.344839    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:56.345335 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:56.344935    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlx9j" podUID="db50e6b2-f690-4c96-a2f6-1843a40e95ec"
Jul 02 13:44:57.027683 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:57.027649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:57.027683 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:57.027690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:57.027873 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:57.027808    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:57.027873 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:57.027868    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret podName:77c36603-0176-42fd-a5db-790d4cb28509 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:29.027855036 +0000 UTC m=+66.223478523 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret") pod "global-pull-secret-syncer-h9rfz" (UID: "77c36603-0176-42fd-a5db-790d4cb28509") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:44:57.027940 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:57.027808    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:57.027940 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:57.027927    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs podName:fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de nodeName:}" failed. No retries permitted until 2026-07-02 13:45:29.027915552 +0000 UTC m=+66.223539051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs") pod "network-metrics-daemon-6vhv6" (UID: "fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:44:57.128913 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:57.128880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mrp\" (UniqueName: \"kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp\") pod \"network-check-target-tlx9j\" (UID: \"db50e6b2-f690-4c96-a2f6-1843a40e95ec\") " pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:57.129059 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:57.129011    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:44:57.129059 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:57.129030    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:44:57.129059 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:57.129040    2574 projected.go:196] Error preparing data for projected volume kube-api-access-p6mrp for pod openshift-network-diagnostics/network-check-target-tlx9j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:57.129224 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:57.129100    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp podName:db50e6b2-f690-4c96-a2f6-1843a40e95ec nodeName:}" failed. No retries permitted until 2026-07-02 13:45:29.129082161 +0000 UTC m=+66.324705669 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6mrp" (UniqueName: "kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp") pod "network-check-target-tlx9j" (UID: "db50e6b2-f690-4c96-a2f6-1843a40e95ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:44:57.345765 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:57.345688    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:57.346220 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:57.345813    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9rfz" podUID="77c36603-0176-42fd-a5db-790d4cb28509"
Jul 02 13:44:57.346220 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:57.345850    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:57.346220 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:57.345934    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6vhv6" podUID="fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de"
Jul 02 13:44:57.966098 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:57.966065    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-214.ec2.internal" event="NodeReady"
Jul 02 13:44:57.966295 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:57.966213    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 13:44:58.023471 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.023436    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9fqv7"]
Jul 02 13:44:58.048371 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.048345    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2knhz"]
Jul 02 13:44:58.048532 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.048480    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.053384 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.053243    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 13:44:58.053384 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.053279    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rwl8t\""
Jul 02 13:44:58.053384 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.053370    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 13:44:58.062526 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.062504    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9fqv7"]
Jul 02 13:44:58.062650 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.062547    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2knhz"]
Jul 02 13:44:58.062650 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.062627    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2knhz"
Jul 02 13:44:58.065356 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.065340    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 13:44:58.066278 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.066260    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 13:44:58.066377 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.066322    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 13:44:58.066430 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.066379    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-srjlp\""
Jul 02 13:44:58.135865 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.135830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.136035 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.135881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17913ead-b221-41db-b8e9-8d3d95c929a6-config-volume\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.136035 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.135928    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4jt8\" (UniqueName: \"kubernetes.io/projected/17913ead-b221-41db-b8e9-8d3d95c929a6-kube-api-access-h4jt8\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.136120 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.136044    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17913ead-b221-41db-b8e9-8d3d95c929a6-tmp-dir\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.236588 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.236494    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17913ead-b221-41db-b8e9-8d3d95c929a6-tmp-dir\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.236588 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.236548    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.236588 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.236583    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17913ead-b221-41db-b8e9-8d3d95c929a6-config-volume\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.236862 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.236613    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fbf0c5d-ae9e-4d73-a309-eb3734f78b36-cert\") pod \"ingress-canary-2knhz\" (UID: \"2fbf0c5d-ae9e-4d73-a309-eb3734f78b36\") " pod="openshift-ingress-canary/ingress-canary-2knhz"
Jul 02 13:44:58.236862 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:58.236731    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:44:58.236862 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.236753    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4jt8\" (UniqueName: \"kubernetes.io/projected/17913ead-b221-41db-b8e9-8d3d95c929a6-kube-api-access-h4jt8\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.236862 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:58.236812    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls podName:17913ead-b221-41db-b8e9-8d3d95c929a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:44:58.736779474 +0000 UTC m=+35.932402961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls") pod "dns-default-9fqv7" (UID: "17913ead-b221-41db-b8e9-8d3d95c929a6") : secret "dns-default-metrics-tls" not found
Jul 02 13:44:58.237039 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.236871    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17913ead-b221-41db-b8e9-8d3d95c929a6-tmp-dir\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.237039 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.236884    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78fll\" (UniqueName: \"kubernetes.io/projected/2fbf0c5d-ae9e-4d73-a309-eb3734f78b36-kube-api-access-78fll\") pod \"ingress-canary-2knhz\" (UID: \"2fbf0c5d-ae9e-4d73-a309-eb3734f78b36\") " pod="openshift-ingress-canary/ingress-canary-2knhz"
Jul 02 13:44:58.237228 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.237192    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17913ead-b221-41db-b8e9-8d3d95c929a6-config-volume\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.249256 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.249219    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4jt8\" (UniqueName: \"kubernetes.io/projected/17913ead-b221-41db-b8e9-8d3d95c929a6-kube-api-access-h4jt8\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.337846 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.337806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fbf0c5d-ae9e-4d73-a309-eb3734f78b36-cert\") pod \"ingress-canary-2knhz\" (UID: \"2fbf0c5d-ae9e-4d73-a309-eb3734f78b36\") " pod="openshift-ingress-canary/ingress-canary-2knhz"
Jul 02 13:44:58.338013 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.337906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78fll\" (UniqueName: \"kubernetes.io/projected/2fbf0c5d-ae9e-4d73-a309-eb3734f78b36-kube-api-access-78fll\") pod \"ingress-canary-2knhz\" (UID: \"2fbf0c5d-ae9e-4d73-a309-eb3734f78b36\") " pod="openshift-ingress-canary/ingress-canary-2knhz"
Jul 02 13:44:58.340601 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.340571    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fbf0c5d-ae9e-4d73-a309-eb3734f78b36-cert\") pod \"ingress-canary-2knhz\" (UID: \"2fbf0c5d-ae9e-4d73-a309-eb3734f78b36\") " pod="openshift-ingress-canary/ingress-canary-2knhz"
Jul 02 13:44:58.344814 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.344781    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:44:58.350972 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.350945    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:44:58.351529 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.351002    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:44:58.351529 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.351028    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bsszs\""
Jul 02 13:44:58.354108 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.354085    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78fll\" (UniqueName: \"kubernetes.io/projected/2fbf0c5d-ae9e-4d73-a309-eb3734f78b36-kube-api-access-78fll\") pod \"ingress-canary-2knhz\" (UID: \"2fbf0c5d-ae9e-4d73-a309-eb3734f78b36\") " pod="openshift-ingress-canary/ingress-canary-2knhz"
Jul 02 13:44:58.372165 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.372125    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2knhz"
Jul 02 13:44:58.741213 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.741179    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:58.741387 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:58.741366    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:44:58.741460 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:58.741449    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls podName:17913ead-b221-41db-b8e9-8d3d95c929a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:44:59.741425774 +0000 UTC m=+36.937049279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls") pod "dns-default-9fqv7" (UID: "17913ead-b221-41db-b8e9-8d3d95c929a6") : secret "dns-default-metrics-tls" not found
Jul 02 13:44:58.811460 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:58.811432    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2knhz"]
Jul 02 13:44:58.859041 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:44:58.859008    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbf0c5d_ae9e_4d73_a309_eb3734f78b36.slice/crio-c101d5a75eb1118026b7596bccfa870e1501073daf68083255d821d486fc9c0b WatchSource:0}: Error finding container c101d5a75eb1118026b7596bccfa870e1501073daf68083255d821d486fc9c0b: Status 404 returned error can't find the container with id c101d5a75eb1118026b7596bccfa870e1501073daf68083255d821d486fc9c0b
Jul 02 13:44:59.345021 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:59.344796    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:44:59.345190 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:59.344824    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:44:59.348090 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:59.348034    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:44:59.348090 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:59.348043    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f4vwh\""
Jul 02 13:44:59.349016 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:59.349000    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:44:59.630013 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:59.629925    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2knhz" event={"ID":"2fbf0c5d-ae9e-4d73-a309-eb3734f78b36","Type":"ContainerStarted","Data":"c101d5a75eb1118026b7596bccfa870e1501073daf68083255d821d486fc9c0b"}
Jul 02 13:44:59.632891 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:59.632860    2574 generic.go:356] "Generic (PLEG): container finished" podID="be6b5fc4-8ec4-4bf6-8386-9918c653640a" containerID="0d6ac42e3877adfce970b7951d2ab1b76f28da2eec669bafb9a4aac12d30ec51" exitCode=0
Jul 02 13:44:59.633016 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:59.632914    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvh8f" event={"ID":"be6b5fc4-8ec4-4bf6-8386-9918c653640a","Type":"ContainerDied","Data":"0d6ac42e3877adfce970b7951d2ab1b76f28da2eec669bafb9a4aac12d30ec51"}
Jul 02 13:44:59.748377 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:44:59.748345    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:44:59.748542 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:59.748522    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:44:59.748605 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:44:59.748590    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls podName:17913ead-b221-41db-b8e9-8d3d95c929a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:01.748570546 +0000 UTC m=+38.944194034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls") pod "dns-default-9fqv7" (UID: "17913ead-b221-41db-b8e9-8d3d95c929a6") : secret "dns-default-metrics-tls" not found
Jul 02 13:45:00.638834 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:00.638796    2574 generic.go:356] "Generic (PLEG): container finished" podID="be6b5fc4-8ec4-4bf6-8386-9918c653640a" containerID="c27d6bf2ef2c250cf2d5beb918d4d6fbd3bd690fe936fa9a227178092e64e706" exitCode=0
Jul 02 13:45:00.639235 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:00.638846    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvh8f" event={"ID":"be6b5fc4-8ec4-4bf6-8386-9918c653640a","Type":"ContainerDied","Data":"c27d6bf2ef2c250cf2d5beb918d4d6fbd3bd690fe936fa9a227178092e64e706"}
Jul 02 13:45:01.642338 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:01.642304    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2knhz" event={"ID":"2fbf0c5d-ae9e-4d73-a309-eb3734f78b36","Type":"ContainerStarted","Data":"5b4bac5119f6030302634f8bb3e5169a833d1d493db5f559b92de4b29412dc39"}
Jul 02 13:45:01.645380 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:01.645354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wvh8f" event={"ID":"be6b5fc4-8ec4-4bf6-8386-9918c653640a","Type":"ContainerStarted","Data":"bc3e076976f87e304e864f52ed84b34ad40a75d5c1a1127caba31475f025b526"}
Jul 02 13:45:01.668594 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:01.668542    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2knhz" podStartSLOduration=1.303714348 podStartE2EDuration="3.668521935s" totalImagesPullingTime="2.364807587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:01.667753101 +0000 UTC m=+38.863376609" watchObservedRunningTime="2026-07-02 13:45:01.668521935 +0000 UTC m=+38.864145449"
Jul 02 13:45:01.707154 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:01.705228    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wvh8f" podStartSLOduration=8.881680306 podStartE2EDuration="38.70512639s" totalImagesPullingTime="29.823446084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:01.698588588 +0000 UTC m=+38.894212098" watchObservedRunningTime="2026-07-02 13:45:01.70512639 +0000 UTC m=+38.900749898"
Jul 02 13:45:01.764458 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:01.764414    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:45:01.764664 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:01.764552    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:45:01.764664 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:01.764616    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls podName:17913ead-b221-41db-b8e9-8d3d95c929a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:05.764597948 +0000 UTC m=+42.960221445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls") pod "dns-default-9fqv7" (UID: "17913ead-b221-41db-b8e9-8d3d95c929a6") : secret "dns-default-metrics-tls" not found
Jul 02 13:45:04.280481 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.280449    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"]
Jul 02 13:45:04.282413 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.282395    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"
Jul 02 13:45:04.288819 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.288790    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 02 13:45:04.288945 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.288925    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:45:04.289042 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.289027    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fksqh\""
Jul 02 13:45:04.289107 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.289049    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 02 13:45:04.289107 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.289046    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 02 13:45:04.298752 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.298731    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"]
Jul 02 13:45:04.383681 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.383648    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7vs9\" (UniqueName: \"kubernetes.io/projected/bd56974d-3878-413a-9cb8-b56a9db313fc-kube-api-access-g7vs9\") pod \"service-ca-operator-5d7b7dc95c-djtvf\" (UID: \"bd56974d-3878-413a-9cb8-b56a9db313fc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"
Jul 02 13:45:04.383681 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.383689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd56974d-3878-413a-9cb8-b56a9db313fc-config\") pod \"service-ca-operator-5d7b7dc95c-djtvf\" (UID: \"bd56974d-3878-413a-9cb8-b56a9db313fc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"
Jul 02 13:45:04.383883 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.383715    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd56974d-3878-413a-9cb8-b56a9db313fc-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-djtvf\" (UID: \"bd56974d-3878-413a-9cb8-b56a9db313fc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"
Jul 02 13:45:04.410510 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.410477    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-4frv4"]
Jul 02 13:45:04.414842 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.414827    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.421467 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.421447    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 02 13:45:04.421677 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.421662    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 02 13:45:04.421776 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.421758    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:45:04.421851 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.421822    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4mfp4\""
Jul 02 13:45:04.423466 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.423448    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 02 13:45:04.431067 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.431049    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 02 13:45:04.434358 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.434338    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-4frv4"]
Jul 02 13:45:04.485052 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.485020    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7vs9\" (UniqueName: \"kubernetes.io/projected/bd56974d-3878-413a-9cb8-b56a9db313fc-kube-api-access-g7vs9\") pod \"service-ca-operator-5d7b7dc95c-djtvf\" (UID: \"bd56974d-3878-413a-9cb8-b56a9db313fc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"
Jul 02 13:45:04.485211 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.485063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd56974d-3878-413a-9cb8-b56a9db313fc-config\") pod \"service-ca-operator-5d7b7dc95c-djtvf\" (UID: \"bd56974d-3878-413a-9cb8-b56a9db313fc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"
Jul 02 13:45:04.485211 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.485164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd56974d-3878-413a-9cb8-b56a9db313fc-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-djtvf\" (UID: \"bd56974d-3878-413a-9cb8-b56a9db313fc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"
Jul 02 13:45:04.485584 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.485564    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd56974d-3878-413a-9cb8-b56a9db313fc-config\") pod \"service-ca-operator-5d7b7dc95c-djtvf\" (UID: \"bd56974d-3878-413a-9cb8-b56a9db313fc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"
Jul 02 13:45:04.489110 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.489091    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd56974d-3878-413a-9cb8-b56a9db313fc-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-djtvf\" (UID: \"bd56974d-3878-413a-9cb8-b56a9db313fc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"
Jul 02 13:45:04.501481 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.501460    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-ffd7db6d5-swxmp"]
Jul 02 13:45:04.502370 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.502345    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7vs9\" (UniqueName: \"kubernetes.io/projected/bd56974d-3878-413a-9cb8-b56a9db313fc-kube-api-access-g7vs9\") pod \"service-ca-operator-5d7b7dc95c-djtvf\" (UID: \"bd56974d-3878-413a-9cb8-b56a9db313fc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"
Jul 02 13:45:04.504827 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.504806    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"]
Jul 02 13:45:04.504969 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.504951    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.507631 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.507615    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5f67b55555-s6bjn"]
Jul 02 13:45:04.507762 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.507746    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:04.508244 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.508226    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 13:45:04.508657 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.508639    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j4vz9\""
Jul 02 13:45:04.509630 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.509614    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 13:45:04.511374 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.511357    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.512636 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.512608    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 02 13:45:04.512735 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.512647    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 02 13:45:04.513076 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.513056    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:45:04.513269 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.513253    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rmmfz\""
Jul 02 13:45:04.513434 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.513413    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 13:45:04.514440 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.514422    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 02 13:45:04.515016 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.514999    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 13:45:04.515343 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.515326    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 13:45:04.515343 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.515335    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 13:45:04.515577 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.515557    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7ftkw\""
Jul 02 13:45:04.515666 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.515556    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 02 13:45:04.525167 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.525089    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 02 13:45:04.525380 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.525360    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"]
Jul 02 13:45:04.526475 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.526453    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-s6bjn"]
Jul 02 13:45:04.528025 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.528005    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ffd7db6d5-swxmp"]
Jul 02 13:45:04.585631 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.585533    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed604da1-f64a-4102-8272-8cdf4c804988-config\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.585631 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.585606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfn5b\" (UniqueName: \"kubernetes.io/projected/ed604da1-f64a-4102-8272-8cdf4c804988-kube-api-access-xfn5b\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.585801 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.585690    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed604da1-f64a-4102-8272-8cdf4c804988-serving-cert\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.585801 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.585721    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed604da1-f64a-4102-8272-8cdf4c804988-trusted-ca\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.591461 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.591444    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"
Jul 02 13:45:04.686377 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686343    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed604da1-f64a-4102-8272-8cdf4c804988-serving-cert\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.686547 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686383    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvv9m\" (UniqueName: \"kubernetes.io/projected/b236de76-d2f6-469a-8bed-f59597794af0-kube-api-access-fvv9m\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.686547 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686406    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed604da1-f64a-4102-8272-8cdf4c804988-trusted-ca\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.686547 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686424    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd439b40-ef2a-416e-8475-8c84e660989e-ca-trust-extracted\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.686547 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686477    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-registry-certificates\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.686547 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686504    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b236de76-d2f6-469a-8bed-f59597794af0-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.686547 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686535    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-image-registry-private-configuration\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.686854 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686569    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-trusted-ca\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.686854 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686628    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.686854 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686656    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b236de76-d2f6-469a-8bed-f59597794af0-service-ca-bundle\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.686854 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686681    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bch26\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-kube-api-access-bch26\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.686854 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686708    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-j59cz\" (UID: \"c3390f82-3c55-4c64-b191-a39da63b2d4d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:04.686854 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b236de76-d2f6-469a-8bed-f59597794af0-tmp\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.686854 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686810    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b236de76-d2f6-469a-8bed-f59597794af0-snapshots\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.686854 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.686834    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b236de76-d2f6-469a-8bed-f59597794af0-serving-cert\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.687228 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.687120    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed604da1-f64a-4102-8272-8cdf4c804988-config\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.687228 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.687194    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfn5b\" (UniqueName: \"kubernetes.io/projected/ed604da1-f64a-4102-8272-8cdf4c804988-kube-api-access-xfn5b\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.687329 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.687241    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-bound-sa-token\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.687329 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.687275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-installation-pull-secrets\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.687329 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.687300    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7g6\" (UniqueName: \"kubernetes.io/projected/c3390f82-3c55-4c64-b191-a39da63b2d4d-kube-api-access-xd7g6\") pod \"cluster-samples-operator-9ddc5ff76-j59cz\" (UID: \"c3390f82-3c55-4c64-b191-a39da63b2d4d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:04.687478 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.687352    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed604da1-f64a-4102-8272-8cdf4c804988-trusted-ca\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.687737 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.687716    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed604da1-f64a-4102-8272-8cdf4c804988-config\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.688599 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.688574    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed604da1-f64a-4102-8272-8cdf4c804988-serving-cert\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.696739 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.696716    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfn5b\" (UniqueName: \"kubernetes.io/projected/ed604da1-f64a-4102-8272-8cdf4c804988-kube-api-access-xfn5b\") pod \"console-operator-7fdbdf9c99-4frv4\" (UID: \"ed604da1-f64a-4102-8272-8cdf4c804988\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.723753 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.723721    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:04.737988 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.737963    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf"]
Jul 02 13:45:04.752330 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:04.752309    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd56974d_3878_413a_9cb8_b56a9db313fc.slice/crio-926a6fd16703729d8db98237fac1fa97bfdbb4f3dbe67a195af3b7ca738ea8b5 WatchSource:0}: Error finding container 926a6fd16703729d8db98237fac1fa97bfdbb4f3dbe67a195af3b7ca738ea8b5: Status 404 returned error can't find the container with id 926a6fd16703729d8db98237fac1fa97bfdbb4f3dbe67a195af3b7ca738ea8b5
Jul 02 13:45:04.788003 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.787975    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-registry-certificates\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.788156 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788010    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b236de76-d2f6-469a-8bed-f59597794af0-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.788156 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788033    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-image-registry-private-configuration\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.788156 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788052    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-trusted-ca\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.788156 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788098    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.788156 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b236de76-d2f6-469a-8bed-f59597794af0-service-ca-bundle\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.788405 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bch26\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-kube-api-access-bch26\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.788405 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788190    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-j59cz\" (UID: \"c3390f82-3c55-4c64-b191-a39da63b2d4d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:04.788405 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788222    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b236de76-d2f6-469a-8bed-f59597794af0-tmp\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.788405 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:04.788246    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:45:04.788405 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:04.788269    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ffd7db6d5-swxmp: secret "image-registry-tls" not found
Jul 02 13:45:04.788405 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:04.788334    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls podName:cd439b40-ef2a-416e-8475-8c84e660989e nodeName:}" failed. No retries permitted until 2026-07-02 13:45:05.288311927 +0000 UTC m=+42.483935427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls") pod "image-registry-ffd7db6d5-swxmp" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e") : secret "image-registry-tls" not found
Jul 02 13:45:04.788693 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788255    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b236de76-d2f6-469a-8bed-f59597794af0-snapshots\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.788693 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b236de76-d2f6-469a-8bed-f59597794af0-serving-cert\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.788693 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788527    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-bound-sa-token\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.788693 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-installation-pull-secrets\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.788693 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788592    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7g6\" (UniqueName: \"kubernetes.io/projected/c3390f82-3c55-4c64-b191-a39da63b2d4d-kube-api-access-xd7g6\") pod \"cluster-samples-operator-9ddc5ff76-j59cz\" (UID: \"c3390f82-3c55-4c64-b191-a39da63b2d4d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:04.788693 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvv9m\" (UniqueName: \"kubernetes.io/projected/b236de76-d2f6-469a-8bed-f59597794af0-kube-api-access-fvv9m\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.788693 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd439b40-ef2a-416e-8475-8c84e660989e-ca-trust-extracted\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.788954 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788831    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b236de76-d2f6-469a-8bed-f59597794af0-snapshots\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.788954 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788872    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b236de76-d2f6-469a-8bed-f59597794af0-service-ca-bundle\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.788954 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:04.788919    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 13:45:04.789063 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.788948    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b236de76-d2f6-469a-8bed-f59597794af0-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.789063 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:04.788968    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls podName:c3390f82-3c55-4c64-b191-a39da63b2d4d nodeName:}" failed. No retries permitted until 2026-07-02 13:45:05.28895291 +0000 UTC m=+42.484576415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-j59cz" (UID: "c3390f82-3c55-4c64-b191-a39da63b2d4d") : secret "samples-operator-tls" not found
Jul 02 13:45:04.789209 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.789181    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b236de76-d2f6-469a-8bed-f59597794af0-tmp\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.789460 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.789431    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd439b40-ef2a-416e-8475-8c84e660989e-ca-trust-extracted\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.789734 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.789711    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-registry-certificates\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.790349 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.790324    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-trusted-ca\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.791054 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.791033    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b236de76-d2f6-469a-8bed-f59597794af0-serving-cert\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.791206 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.791186    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-image-registry-private-configuration\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.791328 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.791311    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-installation-pull-secrets\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.800246 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.799822    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bch26\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-kube-api-access-bch26\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.800246 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.800060    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-bound-sa-token\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:04.800422 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.800253    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7g6\" (UniqueName: \"kubernetes.io/projected/c3390f82-3c55-4c64-b191-a39da63b2d4d-kube-api-access-xd7g6\") pod \"cluster-samples-operator-9ddc5ff76-j59cz\" (UID: \"c3390f82-3c55-4c64-b191-a39da63b2d4d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:04.800422 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.800349    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvv9m\" (UniqueName: \"kubernetes.io/projected/b236de76-d2f6-469a-8bed-f59597794af0-kube-api-access-fvv9m\") pod \"insights-operator-5f67b55555-s6bjn\" (UID: \"b236de76-d2f6-469a-8bed-f59597794af0\") " pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.833606 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.833581    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-s6bjn"
Jul 02 13:45:04.871111 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.871086    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-4frv4"]
Jul 02 13:45:04.875539 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:04.875511    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded604da1_f64a_4102_8272_8cdf4c804988.slice/crio-39e2cea291ecedeac65181ea45b5ee54076fa2e9f49d068e6c8a70c0973262e2 WatchSource:0}: Error finding container 39e2cea291ecedeac65181ea45b5ee54076fa2e9f49d068e6c8a70c0973262e2: Status 404 returned error can't find the container with id 39e2cea291ecedeac65181ea45b5ee54076fa2e9f49d068e6c8a70c0973262e2
Jul 02 13:45:04.973450 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:04.973417    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-s6bjn"]
Jul 02 13:45:04.975952 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:04.975928    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb236de76_d2f6_469a_8bed_f59597794af0.slice/crio-ba93bf38efd4af85c0487c03976579d6efae2b49553bf76b5bb0af551d375ce4 WatchSource:0}: Error finding container ba93bf38efd4af85c0487c03976579d6efae2b49553bf76b5bb0af551d375ce4: Status 404 returned error can't find the container with id ba93bf38efd4af85c0487c03976579d6efae2b49553bf76b5bb0af551d375ce4
Jul 02 13:45:05.293378 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:05.293342    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:05.293378 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:05.293381    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-j59cz\" (UID: \"c3390f82-3c55-4c64-b191-a39da63b2d4d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:05.293835 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:05.293537    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:45:05.293835 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:05.293548    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 13:45:05.293835 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:05.293559    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ffd7db6d5-swxmp: secret "image-registry-tls" not found
Jul 02 13:45:05.293835 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:05.293603    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls podName:c3390f82-3c55-4c64-b191-a39da63b2d4d nodeName:}" failed. No retries permitted until 2026-07-02 13:45:06.293588845 +0000 UTC m=+43.489212329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-j59cz" (UID: "c3390f82-3c55-4c64-b191-a39da63b2d4d") : secret "samples-operator-tls" not found
Jul 02 13:45:05.293835 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:05.293616    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls podName:cd439b40-ef2a-416e-8475-8c84e660989e nodeName:}" failed. No retries permitted until 2026-07-02 13:45:06.293609836 +0000 UTC m=+43.489233321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls") pod "image-registry-ffd7db6d5-swxmp" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e") : secret "image-registry-tls" not found
Jul 02 13:45:05.656184 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:05.656077    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4" event={"ID":"ed604da1-f64a-4102-8272-8cdf4c804988","Type":"ContainerStarted","Data":"39e2cea291ecedeac65181ea45b5ee54076fa2e9f49d068e6c8a70c0973262e2"}
Jul 02 13:45:05.657612 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:05.657574    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-s6bjn" event={"ID":"b236de76-d2f6-469a-8bed-f59597794af0","Type":"ContainerStarted","Data":"ba93bf38efd4af85c0487c03976579d6efae2b49553bf76b5bb0af551d375ce4"}
Jul 02 13:45:05.658963 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:05.658937    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf" event={"ID":"bd56974d-3878-413a-9cb8-b56a9db313fc","Type":"ContainerStarted","Data":"926a6fd16703729d8db98237fac1fa97bfdbb4f3dbe67a195af3b7ca738ea8b5"}
Jul 02 13:45:05.797637 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:05.797602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:45:05.797806 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:05.797776    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:45:05.797873 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:05.797855    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls podName:17913ead-b221-41db-b8e9-8d3d95c929a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:13.797831947 +0000 UTC m=+50.993455451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls") pod "dns-default-9fqv7" (UID: "17913ead-b221-41db-b8e9-8d3d95c929a6") : secret "dns-default-metrics-tls" not found
Jul 02 13:45:06.303117 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:06.303077    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:06.303557 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:06.303150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-j59cz\" (UID: \"c3390f82-3c55-4c64-b191-a39da63b2d4d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:06.303557 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:06.303272    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 13:45:06.303557 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:06.303282    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:45:06.303557 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:06.303305    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ffd7db6d5-swxmp: secret "image-registry-tls" not found
Jul 02 13:45:06.303557 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:06.303341    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls podName:c3390f82-3c55-4c64-b191-a39da63b2d4d nodeName:}" failed. No retries permitted until 2026-07-02 13:45:08.303321911 +0000 UTC m=+45.498945398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-j59cz" (UID: "c3390f82-3c55-4c64-b191-a39da63b2d4d") : secret "samples-operator-tls" not found
Jul 02 13:45:06.303557 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:06.303359    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls podName:cd439b40-ef2a-416e-8475-8c84e660989e nodeName:}" failed. No retries permitted until 2026-07-02 13:45:08.303350244 +0000 UTC m=+45.498973735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls") pod "image-registry-ffd7db6d5-swxmp" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e") : secret "image-registry-tls" not found
Jul 02 13:45:08.318707 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.318665    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:08.319202 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.318720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-j59cz\" (UID: \"c3390f82-3c55-4c64-b191-a39da63b2d4d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:08.319202 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:08.318864    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 13:45:08.319202 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:08.318869    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:45:08.319202 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:08.318893    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ffd7db6d5-swxmp: secret "image-registry-tls" not found
Jul 02 13:45:08.319202 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:08.318939    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls podName:c3390f82-3c55-4c64-b191-a39da63b2d4d nodeName:}" failed. No retries permitted until 2026-07-02 13:45:12.31892269 +0000 UTC m=+49.514546188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-j59cz" (UID: "c3390f82-3c55-4c64-b191-a39da63b2d4d") : secret "samples-operator-tls" not found
Jul 02 13:45:08.319202 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:08.318959    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls podName:cd439b40-ef2a-416e-8475-8c84e660989e nodeName:}" failed. No retries permitted until 2026-07-02 13:45:12.31894956 +0000 UTC m=+49.514573046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls") pod "image-registry-ffd7db6d5-swxmp" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e") : secret "image-registry-tls" not found
Jul 02 13:45:08.665623 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.665597    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt"]
Jul 02 13:45:08.678168 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.678127    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt"
Jul 02 13:45:08.681055 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.681021    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt"]
Jul 02 13:45:08.681312 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.681288    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 02 13:45:08.681425 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.681288    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jgrzw\""
Jul 02 13:45:08.682151 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.682120    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 02 13:45:08.721322 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.721291    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drl7p\" (UniqueName: \"kubernetes.io/projected/2548bdc9-0467-4364-b2ea-44b75e5ca948-kube-api-access-drl7p\") pod \"migrator-786f9fbc5-nmstt\" (UID: \"2548bdc9-0467-4364-b2ea-44b75e5ca948\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt"
Jul 02 13:45:08.822501 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.822414    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drl7p\" (UniqueName: \"kubernetes.io/projected/2548bdc9-0467-4364-b2ea-44b75e5ca948-kube-api-access-drl7p\") pod \"migrator-786f9fbc5-nmstt\" (UID: \"2548bdc9-0467-4364-b2ea-44b75e5ca948\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt"
Jul 02 13:45:08.834813 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.834777    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drl7p\" (UniqueName: \"kubernetes.io/projected/2548bdc9-0467-4364-b2ea-44b75e5ca948-kube-api-access-drl7p\") pod \"migrator-786f9fbc5-nmstt\" (UID: \"2548bdc9-0467-4364-b2ea-44b75e5ca948\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt"
Jul 02 13:45:08.988252 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:08.988220    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt"
Jul 02 13:45:09.123331 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.123299    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt"]
Jul 02 13:45:09.127293 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:09.127272    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2548bdc9_0467_4364_b2ea_44b75e5ca948.slice/crio-22e6abd356172d429f607632f2b5f3b3e331abf7bd2dddce1a0072a658142353 WatchSource:0}: Error finding container 22e6abd356172d429f607632f2b5f3b3e331abf7bd2dddce1a0072a658142353: Status 404 returned error can't find the container with id 22e6abd356172d429f607632f2b5f3b3e331abf7bd2dddce1a0072a658142353
Jul 02 13:45:09.670073 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.670042    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/0.log"
Jul 02 13:45:09.670522 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.670088    2574 generic.go:356] "Generic (PLEG): container finished" podID="ed604da1-f64a-4102-8272-8cdf4c804988" containerID="beb5f951de112d6a34775e3c422d6b376a94cdc2811f7c10a6518dab5374370f" exitCode=255
Jul 02 13:45:09.670522 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.670170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4" event={"ID":"ed604da1-f64a-4102-8272-8cdf4c804988","Type":"ContainerDied","Data":"beb5f951de112d6a34775e3c422d6b376a94cdc2811f7c10a6518dab5374370f"}
Jul 02 13:45:09.670522 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.670476    2574 scope.go:117] "RemoveContainer" containerID="beb5f951de112d6a34775e3c422d6b376a94cdc2811f7c10a6518dab5374370f"
Jul 02 13:45:09.671459 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.671436    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt" event={"ID":"2548bdc9-0467-4364-b2ea-44b75e5ca948","Type":"ContainerStarted","Data":"22e6abd356172d429f607632f2b5f3b3e331abf7bd2dddce1a0072a658142353"}
Jul 02 13:45:09.672912 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.672891    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-s6bjn" event={"ID":"b236de76-d2f6-469a-8bed-f59597794af0","Type":"ContainerStarted","Data":"1a426b27b0219dbb209d2e0068938278d5d927112f2ac7f082d35895da643796"}
Jul 02 13:45:09.674457 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.674437    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf" event={"ID":"bd56974d-3878-413a-9cb8-b56a9db313fc","Type":"ContainerStarted","Data":"c0c784d7440f5ccdfbefd975c31bdad6b8f10881d12ed0736ec732728deab027"}
Jul 02 13:45:09.709153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.709098    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf" podStartSLOduration=1.7631269999999999 podStartE2EDuration="5.709083524s" totalImagesPullingTime="3.945956524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:09.708439127 +0000 UTC m=+46.904062658" watchObservedRunningTime="2026-07-02 13:45:09.709083524 +0000 UTC m=+46.904707008"
Jul 02 13:45:09.736171 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.734836    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5f67b55555-s6bjn" podStartSLOduration=2.012512051 podStartE2EDuration="5.734816332s" totalImagesPullingTime="3.722304281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:09.732817405 +0000 UTC m=+46.928440913" watchObservedRunningTime="2026-07-02 13:45:09.734816332 +0000 UTC m=+46.930439840"
Jul 02 13:45:09.960283 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.960209    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n8mlp"]
Jul 02 13:45:09.983850 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.983808    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n8mlp"]
Jul 02 13:45:09.984015 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.983955    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:09.987110 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.987086    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 13:45:09.987262 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.987112    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 13:45:09.987262 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:09.987110    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k62kw\""
Jul 02 13:45:10.034012 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.033977    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8bm\" (UniqueName: \"kubernetes.io/projected/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-api-access-8f8bm\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.034211 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.034045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.034211 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.034072    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.034211 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.034125    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e76678b-b568-49d1-8c10-78d92083fbd9-crio-socket\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.034211 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.034193    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e76678b-b568-49d1-8c10-78d92083fbd9-data-volume\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.134952 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.134915    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e76678b-b568-49d1-8c10-78d92083fbd9-data-volume\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.135160 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.135016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8bm\" (UniqueName: \"kubernetes.io/projected/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-api-access-8f8bm\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.135160 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.135068    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.135160 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.135100    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.135333 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.135182    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e76678b-b568-49d1-8c10-78d92083fbd9-crio-socket\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.135333 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:10.135194    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:45:10.135333 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:10.135293    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm podName:7e76678b-b568-49d1-8c10-78d92083fbd9 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:10.635270853 +0000 UTC m=+47.830894350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n8mlp" (UID: "7e76678b-b568-49d1-8c10-78d92083fbd9") : configmap "kube-rbac-proxy" not found
Jul 02 13:45:10.135492 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.135331    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e76678b-b568-49d1-8c10-78d92083fbd9-data-volume\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.135492 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.135384    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e76678b-b568-49d1-8c10-78d92083fbd9-crio-socket\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.135492 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:10.135457    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:10.135629 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:10.135506    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls podName:7e76678b-b568-49d1-8c10-78d92083fbd9 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:10.635489035 +0000 UTC m=+47.831112521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n8mlp" (UID: "7e76678b-b568-49d1-8c10-78d92083fbd9") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:10.150900 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.150871    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8bm\" (UniqueName: \"kubernetes.io/projected/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-api-access-8f8bm\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.640172 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.640126    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.640172 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.640179    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:10.640383 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:10.640256    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:45:10.640383 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:10.640337    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm podName:7e76678b-b568-49d1-8c10-78d92083fbd9 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:11.640319196 +0000 UTC m=+48.835942694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n8mlp" (UID: "7e76678b-b568-49d1-8c10-78d92083fbd9") : configmap "kube-rbac-proxy" not found
Jul 02 13:45:10.640383 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:10.640357    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:10.640511 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:10.640419    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls podName:7e76678b-b568-49d1-8c10-78d92083fbd9 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:11.640406057 +0000 UTC m=+48.836029545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n8mlp" (UID: "7e76678b-b568-49d1-8c10-78d92083fbd9") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:10.679612 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.679588    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 13:45:10.680027 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.680014    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/0.log"
Jul 02 13:45:10.680071 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.680045    2574 generic.go:356] "Generic (PLEG): container finished" podID="ed604da1-f64a-4102-8272-8cdf4c804988" containerID="0615a1a084099d6bd1c217378ce1352cbf284d69bd81945c674d05407c912d2c" exitCode=255
Jul 02 13:45:10.680173 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.680124    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4" event={"ID":"ed604da1-f64a-4102-8272-8cdf4c804988","Type":"ContainerDied","Data":"0615a1a084099d6bd1c217378ce1352cbf284d69bd81945c674d05407c912d2c"}
Jul 02 13:45:10.680237 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.680225    2574 scope.go:117] "RemoveContainer" containerID="beb5f951de112d6a34775e3c422d6b376a94cdc2811f7c10a6518dab5374370f"
Jul 02 13:45:10.680324 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:10.680305    2574 scope.go:117] "RemoveContainer" containerID="0615a1a084099d6bd1c217378ce1352cbf284d69bd81945c674d05407c912d2c"
Jul 02 13:45:10.680475 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:10.680455    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-4frv4_openshift-console-operator(ed604da1-f64a-4102-8272-8cdf4c804988)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4" podUID="ed604da1-f64a-4102-8272-8cdf4c804988"
Jul 02 13:45:11.648328 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:11.648297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:11.648504 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:11.648335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:11.648504 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:11.648417    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:45:11.648504 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:11.648445    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:11.648504 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:11.648475    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm podName:7e76678b-b568-49d1-8c10-78d92083fbd9 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:13.648460892 +0000 UTC m=+50.844084381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n8mlp" (UID: "7e76678b-b568-49d1-8c10-78d92083fbd9") : configmap "kube-rbac-proxy" not found
Jul 02 13:45:11.648504 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:11.648498    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls podName:7e76678b-b568-49d1-8c10-78d92083fbd9 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:13.648480298 +0000 UTC m=+50.844103798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n8mlp" (UID: "7e76678b-b568-49d1-8c10-78d92083fbd9") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:11.685673 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:11.685651    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 13:45:11.686027 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:11.685990    2574 scope.go:117] "RemoveContainer" containerID="0615a1a084099d6bd1c217378ce1352cbf284d69bd81945c674d05407c912d2c"
Jul 02 13:45:11.686179 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:11.686160    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-4frv4_openshift-console-operator(ed604da1-f64a-4102-8272-8cdf4c804988)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4" podUID="ed604da1-f64a-4102-8272-8cdf4c804988"
Jul 02 13:45:11.691161 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:11.691119    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt" event={"ID":"2548bdc9-0467-4364-b2ea-44b75e5ca948","Type":"ContainerStarted","Data":"8151808f143af896b99f58eb1dfac1ae54349ae11a7b549b0e2de0e15a878727"}
Jul 02 13:45:11.691255 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:11.691199    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt" event={"ID":"2548bdc9-0467-4364-b2ea-44b75e5ca948","Type":"ContainerStarted","Data":"7286536a6146147b0b70aed4e257a00e899e05fdd5f42954a4d9f1c6f8b06d0e"}
Jul 02 13:45:11.719723 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:11.719583    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-nmstt" podStartSLOduration=1.789125633 podStartE2EDuration="3.719566791s" totalImagesPullingTime="1.930441158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:11.719319958 +0000 UTC m=+48.914943465" watchObservedRunningTime="2026-07-02 13:45:11.719566791 +0000 UTC m=+48.915190295"
Jul 02 13:45:12.355400 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:12.355361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:12.355400 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:12.355403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-j59cz\" (UID: \"c3390f82-3c55-4c64-b191-a39da63b2d4d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:12.355605 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:12.355512    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:45:12.355605 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:12.355520    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 13:45:12.355605 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:12.355527    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ffd7db6d5-swxmp: secret "image-registry-tls" not found
Jul 02 13:45:12.355605 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:12.355588    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls podName:cd439b40-ef2a-416e-8475-8c84e660989e nodeName:}" failed. No retries permitted until 2026-07-02 13:45:20.355570846 +0000 UTC m=+57.551194345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls") pod "image-registry-ffd7db6d5-swxmp" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e") : secret "image-registry-tls" not found
Jul 02 13:45:12.355605 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:12.355601    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls podName:c3390f82-3c55-4c64-b191-a39da63b2d4d nodeName:}" failed. No retries permitted until 2026-07-02 13:45:20.355595932 +0000 UTC m=+57.551219420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-j59cz" (UID: "c3390f82-3c55-4c64-b191-a39da63b2d4d") : secret "samples-operator-tls" not found
Jul 02 13:45:12.933908 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:12.933882    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sdckb_fac51af3-0564-4b8c-946d-7ce5d5bbdf7e/dns-node-resolver/0.log"
Jul 02 13:45:13.539681 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:13.539652    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d4gsb_fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c/node-ca/0.log"
Jul 02 13:45:13.664762 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:13.664725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:13.664920 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:13.664768    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:13.664920 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:13.664858    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:45:13.665038 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:13.664925    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm podName:7e76678b-b568-49d1-8c10-78d92083fbd9 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:17.664907357 +0000 UTC m=+54.860530861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n8mlp" (UID: "7e76678b-b568-49d1-8c10-78d92083fbd9") : configmap "kube-rbac-proxy" not found
Jul 02 13:45:13.665038 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:13.664983    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:13.665174 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:13.665044    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls podName:7e76678b-b568-49d1-8c10-78d92083fbd9 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:17.665029339 +0000 UTC m=+54.860652846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n8mlp" (UID: "7e76678b-b568-49d1-8c10-78d92083fbd9") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:13.866739 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:13.866661    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:45:13.866871 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:13.866817    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:45:13.866913 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:13.866886    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls podName:17913ead-b221-41db-b8e9-8d3d95c929a6 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:29.866868833 +0000 UTC m=+67.062492332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls") pod "dns-default-9fqv7" (UID: "17913ead-b221-41db-b8e9-8d3d95c929a6") : secret "dns-default-metrics-tls" not found
Jul 02 13:45:14.330792 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:14.330762    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2knhz_2fbf0c5d-ae9e-4d73-a309-eb3734f78b36/serve-healthcheck-canary/0.log"
Jul 02 13:45:14.724129 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:14.724095    2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:14.724333 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:14.724225    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:14.724553 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:14.724540    2574 scope.go:117] "RemoveContainer" containerID="0615a1a084099d6bd1c217378ce1352cbf284d69bd81945c674d05407c912d2c"
Jul 02 13:45:14.724699 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:14.724680    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-4frv4_openshift-console-operator(ed604da1-f64a-4102-8272-8cdf4c804988)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4" podUID="ed604da1-f64a-4102-8272-8cdf4c804988"
Jul 02 13:45:14.937306 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:14.937271    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-nmstt_2548bdc9-0467-4364-b2ea-44b75e5ca948/migrator/0.log"
Jul 02 13:45:15.136390 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:15.136323    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-nmstt_2548bdc9-0467-4364-b2ea-44b75e5ca948/graceful-termination/0.log"
Jul 02 13:45:17.697321 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:17.697283    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:17.697321 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:17.697326    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:17.697809 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:17.697413    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:45:17.697809 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:17.697447    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:17.697809 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:17.697485    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm podName:7e76678b-b568-49d1-8c10-78d92083fbd9 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:25.697469385 +0000 UTC m=+62.893092869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n8mlp" (UID: "7e76678b-b568-49d1-8c10-78d92083fbd9") : configmap "kube-rbac-proxy" not found
Jul 02 13:45:17.697809 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:17.697501    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls podName:7e76678b-b568-49d1-8c10-78d92083fbd9 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:25.697495368 +0000 UTC m=+62.893118853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n8mlp" (UID: "7e76678b-b568-49d1-8c10-78d92083fbd9") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:20.421206 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:20.421168    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-j59cz\" (UID: \"c3390f82-3c55-4c64-b191-a39da63b2d4d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:20.421748 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:20.421361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:20.423681 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:20.423646    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c3390f82-3c55-4c64-b191-a39da63b2d4d-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-j59cz\" (UID: \"c3390f82-3c55-4c64-b191-a39da63b2d4d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:20.423762 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:20.423689    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls\") pod \"image-registry-ffd7db6d5-swxmp\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") " pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:20.427493 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:20.427471    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"
Jul 02 13:45:20.577781 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:20.577751    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz"]
Jul 02 13:45:20.716446 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:20.716381    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:20.717391 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:20.717370    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz" event={"ID":"c3390f82-3c55-4c64-b191-a39da63b2d4d","Type":"ContainerStarted","Data":"eb4820cdaf6abd4ac253a7b17ee4d898d555b25fc3ac623522ed90082644367f"}
Jul 02 13:45:20.853236 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:20.853216    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ffd7db6d5-swxmp"]
Jul 02 13:45:20.855300 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:20.855279    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd439b40_ef2a_416e_8475_8c84e660989e.slice/crio-7bc2a7f4f28291f5ade6d082e950b2c8d7f289fa651c7599415c52f7efaaa86d WatchSource:0}: Error finding container 7bc2a7f4f28291f5ade6d082e950b2c8d7f289fa651c7599415c52f7efaaa86d: Status 404 returned error can't find the container with id 7bc2a7f4f28291f5ade6d082e950b2c8d7f289fa651c7599415c52f7efaaa86d
Jul 02 13:45:21.722527 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:21.722494    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp" event={"ID":"cd439b40-ef2a-416e-8475-8c84e660989e","Type":"ContainerStarted","Data":"67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231"}
Jul 02 13:45:21.722880 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:21.722541    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp" event={"ID":"cd439b40-ef2a-416e-8475-8c84e660989e","Type":"ContainerStarted","Data":"7bc2a7f4f28291f5ade6d082e950b2c8d7f289fa651c7599415c52f7efaaa86d"}
Jul 02 13:45:21.722880 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:21.722634    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:21.743174 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:21.743108    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp" podStartSLOduration=17.743092362 podStartE2EDuration="17.743092362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:21.742347682 +0000 UTC m=+58.937971188" watchObservedRunningTime="2026-07-02 13:45:21.743092362 +0000 UTC m=+58.938715860"
Jul 02 13:45:22.726980 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:22.726903    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz" event={"ID":"c3390f82-3c55-4c64-b191-a39da63b2d4d","Type":"ContainerStarted","Data":"fd3e3206d404446b1a048dc6223226425c830ec71dbe371459128fa3975982e9"}
Jul 02 13:45:22.726980 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:22.726956    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz" event={"ID":"c3390f82-3c55-4c64-b191-a39da63b2d4d","Type":"ContainerStarted","Data":"9a9ba2e64a890eb27943e7cde2df3acb4b0e13d962020b1f42f9e8af6bd0d7fa"}
Jul 02 13:45:22.751889 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:22.751836    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-j59cz" podStartSLOduration=16.953561241 podStartE2EDuration="18.751822315s" totalImagesPullingTime="1.798261074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:22.750535489 +0000 UTC m=+59.946159007" watchObservedRunningTime="2026-07-02 13:45:22.751822315 +0000 UTC m=+59.947445822"
Jul 02 13:45:23.621746 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:23.621718    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lmvr2"
Jul 02 13:45:25.767285 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:25.767258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:25.767285 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:25.767285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:25.767807 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:25.767389    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:25.767807 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:25.767435    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls podName:7e76678b-b568-49d1-8c10-78d92083fbd9 nodeName:}" failed. No retries permitted until 2026-07-02 13:45:41.767422916 +0000 UTC m=+78.963046401 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n8mlp" (UID: "7e76678b-b568-49d1-8c10-78d92083fbd9") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:45:25.767929 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:25.767875    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e76678b-b568-49d1-8c10-78d92083fbd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:27.345357 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:27.345323    2574 scope.go:117] "RemoveContainer" containerID="0615a1a084099d6bd1c217378ce1352cbf284d69bd81945c674d05407c912d2c"
Jul 02 13:45:27.743908 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:27.743876    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 13:45:27.744091 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:27.743954    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4" event={"ID":"ed604da1-f64a-4102-8272-8cdf4c804988","Type":"ContainerStarted","Data":"92201b8b91716402cf603b2c9e545c356b8fd221544379e0ba9922be478c880d"}
Jul 02 13:45:27.744270 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:27.744250    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:27.748844 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:27.748821    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4"
Jul 02 13:45:27.791095 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:27.791043    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7fdbdf9c99-4frv4" podStartSLOduration=19.963085024 podStartE2EDuration="23.791026589s" totalImagesPullingTime="3.827941565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:27.789690211 +0000 UTC m=+64.985313720" watchObservedRunningTime="2026-07-02 13:45:27.791026589 +0000 UTC m=+64.986650097"
Jul 02 13:45:29.093323 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.093277    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:45:29.093712 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.093372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:45:29.096161 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.096129    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:45:29.096763 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.096747    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:45:29.105825 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.105796    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77c36603-0176-42fd-a5db-790d4cb28509-original-pull-secret\") pod \"global-pull-secret-syncer-h9rfz\" (UID: \"77c36603-0176-42fd-a5db-790d4cb28509\") " pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:45:29.105944 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.105803    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de-metrics-certs\") pod \"network-metrics-daemon-6vhv6\" (UID: \"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de\") " pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:45:29.194751 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.194720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mrp\" (UniqueName: \"kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp\") pod \"network-check-target-tlx9j\" (UID: \"db50e6b2-f690-4c96-a2f6-1843a40e95ec\") " pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:45:29.197272 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.197257    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:45:29.207861 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.207847    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:45:29.218092 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.218068    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6mrp\" (UniqueName: \"kubernetes.io/projected/db50e6b2-f690-4c96-a2f6-1843a40e95ec-kube-api-access-p6mrp\") pod \"network-check-target-tlx9j\" (UID: \"db50e6b2-f690-4c96-a2f6-1843a40e95ec\") " pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:45:29.265808 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.265783    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bsszs\""
Jul 02 13:45:29.274076 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.274063    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:45:29.358487 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.358463    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f4vwh\""
Jul 02 13:45:29.360595 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.360575    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9rfz"
Jul 02 13:45:29.365763 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.365741    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6vhv6"
Jul 02 13:45:29.398410 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.398349    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tlx9j"]
Jul 02 13:45:29.494582 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.494549    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h9rfz"]
Jul 02 13:45:29.497588 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:29.497564    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c36603_0176_42fd_a5db_790d4cb28509.slice/crio-e3c89a2ca72bfa145948e9bb4bf2a2242dd759795a049ed3f220e98bcf46c3ed WatchSource:0}: Error finding container e3c89a2ca72bfa145948e9bb4bf2a2242dd759795a049ed3f220e98bcf46c3ed: Status 404 returned error can't find the container with id e3c89a2ca72bfa145948e9bb4bf2a2242dd759795a049ed3f220e98bcf46c3ed
Jul 02 13:45:29.510457 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.510436    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6vhv6"]
Jul 02 13:45:29.513113 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:29.513091    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb870114_0df5_4eb1_bb2a_3ac9e7e2e5de.slice/crio-de70ba50f5da3b5d7724d2bff4430a22336d075153d3f185eb4973d523c736ef WatchSource:0}: Error finding container de70ba50f5da3b5d7724d2bff4430a22336d075153d3f185eb4973d523c736ef: Status 404 returned error can't find the container with id de70ba50f5da3b5d7724d2bff4430a22336d075153d3f185eb4973d523c736ef
Jul 02 13:45:29.751431 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.751393    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6vhv6" event={"ID":"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de","Type":"ContainerStarted","Data":"de70ba50f5da3b5d7724d2bff4430a22336d075153d3f185eb4973d523c736ef"}
Jul 02 13:45:29.752456 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.752430    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tlx9j" event={"ID":"db50e6b2-f690-4c96-a2f6-1843a40e95ec","Type":"ContainerStarted","Data":"e97c21ec7cfe7a07f4f2b3ef79856afe03d2ab93f8b31adb2ff782b34b3f3783"}
Jul 02 13:45:29.753375 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.753356    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h9rfz" event={"ID":"77c36603-0176-42fd-a5db-790d4cb28509","Type":"ContainerStarted","Data":"e3c89a2ca72bfa145948e9bb4bf2a2242dd759795a049ed3f220e98bcf46c3ed"}
Jul 02 13:45:29.900670 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.900628    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:45:29.903254 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:29.903227    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17913ead-b221-41db-b8e9-8d3d95c929a6-metrics-tls\") pod \"dns-default-9fqv7\" (UID: \"17913ead-b221-41db-b8e9-8d3d95c929a6\") " pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:45:30.162069 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:30.161959    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rwl8t\""
Jul 02 13:45:30.169687 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:30.169458    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:45:30.348704 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:30.348673    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9fqv7"]
Jul 02 13:45:30.764598 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:30.764559    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fqv7" event={"ID":"17913ead-b221-41db-b8e9-8d3d95c929a6","Type":"ContainerStarted","Data":"13678c6f3ce51f55e6811da1762af23d293d4cd5d31773860dfd1979c9e817d1"}
Jul 02 13:45:31.770000 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:31.769954    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6vhv6" event={"ID":"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de","Type":"ContainerStarted","Data":"c62cedb8479327d8cb24a847d9371b8f2582bdc21e69c288a874650e508805ca"}
Jul 02 13:45:31.770431 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:31.770021    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6vhv6" event={"ID":"fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de","Type":"ContainerStarted","Data":"c894e14366e5f3860c63717c2452155e12445ad7848f1ecc2cb758a96cb3aaaf"}
Jul 02 13:45:31.800884 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:31.800817    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6vhv6" podStartSLOduration=67.526331863 podStartE2EDuration="1m8.80079562s" totalImagesPullingTime="1.274463757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:31.798895637 +0000 UTC m=+68.994519169" watchObservedRunningTime="2026-07-02 13:45:31.80079562 +0000 UTC m=+68.996419128"
Jul 02 13:45:35.785351 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:35.785308    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h9rfz" event={"ID":"77c36603-0176-42fd-a5db-790d4cb28509","Type":"ContainerStarted","Data":"e2e510298af71b0a90fc506f15f3173fb287744142924082454f29416d4f307a"}
Jul 02 13:45:35.786746 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:35.786723    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tlx9j" event={"ID":"db50e6b2-f690-4c96-a2f6-1843a40e95ec","Type":"ContainerStarted","Data":"19f422e0090367dd22c08a343e588fd8f247f15d25c17631e819ac40a881300a"}
Jul 02 13:45:35.786860 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:35.786785    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:45:35.788429 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:35.788409    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fqv7" event={"ID":"17913ead-b221-41db-b8e9-8d3d95c929a6","Type":"ContainerStarted","Data":"2aaa0540f7342a4216f21efe2c8eb62cac3f33e25cf02c11408db1034db39c40"}
Jul 02 13:45:35.788508 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:35.788446    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fqv7" event={"ID":"17913ead-b221-41db-b8e9-8d3d95c929a6","Type":"ContainerStarted","Data":"c6863eb4bbb404d9ffc89ad1718957736a23d80d01024d32e09dd8bde8e9e5a5"}
Jul 02 13:45:35.788508 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:35.788468    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:45:35.803928 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:35.803889    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h9rfz" podStartSLOduration=67.127097442 podStartE2EDuration="1m12.803876494s" totalImagesPullingTime="5.676779052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:35.803062688 +0000 UTC m=+72.998686197" watchObservedRunningTime="2026-07-02 13:45:35.803876494 +0000 UTC m=+72.999500002"
Jul 02 13:45:35.824877 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:35.824834    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tlx9j" podStartSLOduration=67.067943824 podStartE2EDuration="1m12.824819625s" totalImagesPullingTime="5.756875801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:35.82412863 +0000 UTC m=+73.019752137" watchObservedRunningTime="2026-07-02 13:45:35.824819625 +0000 UTC m=+73.020443131"
Jul 02 13:45:35.846315 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:35.846272    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9fqv7" podStartSLOduration=33.039718898 podStartE2EDuration="37.846258246s" totalImagesPullingTime="4.806539348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:35.845209474 +0000 UTC m=+73.040832975" watchObservedRunningTime="2026-07-02 13:45:35.846258246 +0000 UTC m=+73.041881753"
Jul 02 13:45:37.429213 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.429179    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-ffd7db6d5-swxmp"]
Jul 02 13:45:37.433486 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.433459    2574 patch_prober.go:28] interesting pod/image-registry-ffd7db6d5-swxmp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 02 13:45:37.433627 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.433505    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp" podUID="cd439b40-ef2a-416e-8475-8c84e660989e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:45:37.469007 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.468980    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg"]
Jul 02 13:45:37.471236 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.471221    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg"
Jul 02 13:45:37.504743 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.504717    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 02 13:45:37.505948 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.505931    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-l9md7\""
Jul 02 13:45:37.506104 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.506084    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 02 13:45:37.569192 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.569170    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ef8fe016-fb99-466d-a800-1603da83e1ff-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-2knkg\" (UID: \"ef8fe016-fb99-466d-a800-1603da83e1ff\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg"
Jul 02 13:45:37.569282 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.569218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8fe016-fb99-466d-a800-1603da83e1ff-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2knkg\" (UID: \"ef8fe016-fb99-466d-a800-1603da83e1ff\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg"
Jul 02 13:45:37.618982 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.618958    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg"]
Jul 02 13:45:37.670291 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.670269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ef8fe016-fb99-466d-a800-1603da83e1ff-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-2knkg\" (UID: \"ef8fe016-fb99-466d-a800-1603da83e1ff\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg"
Jul 02 13:45:37.670419 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.670322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8fe016-fb99-466d-a800-1603da83e1ff-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2knkg\" (UID: \"ef8fe016-fb99-466d-a800-1603da83e1ff\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg"
Jul 02 13:45:37.670969 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.670944    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ef8fe016-fb99-466d-a800-1603da83e1ff-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-2knkg\" (UID: \"ef8fe016-fb99-466d-a800-1603da83e1ff\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg"
Jul 02 13:45:37.672680 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.672663    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ef8fe016-fb99-466d-a800-1603da83e1ff-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-2knkg\" (UID: \"ef8fe016-fb99-466d-a800-1603da83e1ff\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg"
Jul 02 13:45:37.779620 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.779546    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg"
Jul 02 13:45:37.962595 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.962566    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg"]
Jul 02 13:45:37.967003 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:37.966972    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8fe016_fb99_466d_a800_1603da83e1ff.slice/crio-f01aa6b66d5fc033f5611dfc22a1e327a6e0a42c2cf9355f661a89df28de4fad WatchSource:0}: Error finding container f01aa6b66d5fc033f5611dfc22a1e327a6e0a42c2cf9355f661a89df28de4fad: Status 404 returned error can't find the container with id f01aa6b66d5fc033f5611dfc22a1e327a6e0a42c2cf9355f661a89df28de4fad
Jul 02 13:45:37.991182 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.991155    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59f6b9978d-hmh9t"]
Jul 02 13:45:37.994651 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:37.994626    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.038738 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.038679    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59f6b9978d-hmh9t"]
Jul 02 13:45:38.074466 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.074434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a358c67-3896-4451-b806-e0a2fdd40ad1-registry-tls\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.074612 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.074472    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a358c67-3896-4451-b806-e0a2fdd40ad1-image-registry-private-configuration\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.074612 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.074506    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a358c67-3896-4451-b806-e0a2fdd40ad1-trusted-ca\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.074612 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.074538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a358c67-3896-4451-b806-e0a2fdd40ad1-installation-pull-secrets\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.074612 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.074560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b9z7\" (UniqueName: \"kubernetes.io/projected/7a358c67-3896-4451-b806-e0a2fdd40ad1-kube-api-access-6b9z7\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.074612 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.074575    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a358c67-3896-4451-b806-e0a2fdd40ad1-bound-sa-token\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.074612 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.074593    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a358c67-3896-4451-b806-e0a2fdd40ad1-ca-trust-extracted\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.074803 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.074667    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a358c67-3896-4451-b806-e0a2fdd40ad1-registry-certificates\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.175799 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.175764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a358c67-3896-4451-b806-e0a2fdd40ad1-installation-pull-secrets\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.175799 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.175800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b9z7\" (UniqueName: \"kubernetes.io/projected/7a358c67-3896-4451-b806-e0a2fdd40ad1-kube-api-access-6b9z7\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.176024 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.175819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a358c67-3896-4451-b806-e0a2fdd40ad1-bound-sa-token\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.176024 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.175837    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a358c67-3896-4451-b806-e0a2fdd40ad1-ca-trust-extracted\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.176024 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.175862    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a358c67-3896-4451-b806-e0a2fdd40ad1-registry-certificates\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.176024 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.175902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a358c67-3896-4451-b806-e0a2fdd40ad1-registry-tls\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.176024 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.175923    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a358c67-3896-4451-b806-e0a2fdd40ad1-image-registry-private-configuration\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.176024 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.175955    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a358c67-3896-4451-b806-e0a2fdd40ad1-trusted-ca\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.176355 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.176331    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a358c67-3896-4451-b806-e0a2fdd40ad1-ca-trust-extracted\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.176781 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.176757    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a358c67-3896-4451-b806-e0a2fdd40ad1-registry-certificates\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.177241 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.177217    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a358c67-3896-4451-b806-e0a2fdd40ad1-trusted-ca\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.178416 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.178393    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a358c67-3896-4451-b806-e0a2fdd40ad1-installation-pull-secrets\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.178469 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.178421    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a358c67-3896-4451-b806-e0a2fdd40ad1-image-registry-private-configuration\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.178469 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.178462    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a358c67-3896-4451-b806-e0a2fdd40ad1-registry-tls\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.200037 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.200011    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a358c67-3896-4451-b806-e0a2fdd40ad1-bound-sa-token\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.201283 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.201266    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b9z7\" (UniqueName: \"kubernetes.io/projected/7a358c67-3896-4451-b806-e0a2fdd40ad1-kube-api-access-6b9z7\") pod \"image-registry-59f6b9978d-hmh9t\" (UID: \"7a358c67-3896-4451-b806-e0a2fdd40ad1\") " pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.303775 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.303690    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:38.441469 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.441426    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59f6b9978d-hmh9t"]
Jul 02 13:45:38.445637 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:38.445613    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a358c67_3896_4451_b806_e0a2fdd40ad1.slice/crio-9d21042e28ebb07ec14e610fe9287fdcb424cff695c67877581baa6fbe0e1af7 WatchSource:0}: Error finding container 9d21042e28ebb07ec14e610fe9287fdcb424cff695c67877581baa6fbe0e1af7: Status 404 returned error can't find the container with id 9d21042e28ebb07ec14e610fe9287fdcb424cff695c67877581baa6fbe0e1af7
Jul 02 13:45:38.800148 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.800101    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t" event={"ID":"7a358c67-3896-4451-b806-e0a2fdd40ad1","Type":"ContainerStarted","Data":"53d2fba5e95625619645a3cfd744bdc89e667ed9bf234e6a071b65638a1eedf1"}
Jul 02 13:45:38.800322 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.800175    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t" event={"ID":"7a358c67-3896-4451-b806-e0a2fdd40ad1","Type":"ContainerStarted","Data":"9d21042e28ebb07ec14e610fe9287fdcb424cff695c67877581baa6fbe0e1af7"}
Jul 02 13:45:38.801306 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.801280    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg" event={"ID":"ef8fe016-fb99-466d-a800-1603da83e1ff","Type":"ContainerStarted","Data":"f01aa6b66d5fc033f5611dfc22a1e327a6e0a42c2cf9355f661a89df28de4fad"}
Jul 02 13:45:38.830621 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:38.830571    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t" podStartSLOduration=1.830555236 podStartE2EDuration="1.830555236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:38.82936733 +0000 UTC m=+76.024990840" watchObservedRunningTime="2026-07-02 13:45:38.830555236 +0000 UTC m=+76.026178781"
Jul 02 13:45:39.806042 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:39.805953    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg" event={"ID":"ef8fe016-fb99-466d-a800-1603da83e1ff","Type":"ContainerStarted","Data":"f1383b7807933aae7285e42e61aa627bcf4f54b38b6669d8376f90a3331c1d48"}
Jul 02 13:45:39.806601 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:39.806232    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:45:39.822049 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:39.821997    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-2knkg" podStartSLOduration=1.2686542539999999 podStartE2EDuration="2.821981069s" totalImagesPullingTime="1.553326815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:39.821974201 +0000 UTC m=+77.017597708" watchObservedRunningTime="2026-07-02 13:45:39.821981069 +0000 UTC m=+77.017604589"
Jul 02 13:45:41.808296 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:41.808255    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:41.810557 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:41.810527    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e76678b-b568-49d1-8c10-78d92083fbd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8mlp\" (UID: \"7e76678b-b568-49d1-8c10-78d92083fbd9\") " pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:42.098499 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:42.098420    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k62kw\""
Jul 02 13:45:42.106364 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:42.106345    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n8mlp"
Jul 02 13:45:42.264981 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:42.264953    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n8mlp"]
Jul 02 13:45:42.268100 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:42.268076    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e76678b_b568_49d1_8c10_78d92083fbd9.slice/crio-5738e3dbdbbeee3c3c3349b46ab39b1626a0c557ddcd4d7c5126736cb4d96561 WatchSource:0}: Error finding container 5738e3dbdbbeee3c3c3349b46ab39b1626a0c557ddcd4d7c5126736cb4d96561: Status 404 returned error can't find the container with id 5738e3dbdbbeee3c3c3349b46ab39b1626a0c557ddcd4d7c5126736cb4d96561
Jul 02 13:45:42.821056 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:42.821010    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8mlp" event={"ID":"7e76678b-b568-49d1-8c10-78d92083fbd9","Type":"ContainerStarted","Data":"5738e3dbdbbeee3c3c3349b46ab39b1626a0c557ddcd4d7c5126736cb4d96561"}
Jul 02 13:45:45.569768 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.569636    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-b8npm"]
Jul 02 13:45:45.579019 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.578995    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:45.583161 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.582916    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 02 13:45:45.583161 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.582965    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 13:45:45.583161 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.582973    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9786c\""
Jul 02 13:45:45.583161 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.582917    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 13:45:45.583161 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.582917    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 13:45:45.583161 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.582916    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 02 13:45:45.584266 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.584243    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-b8npm"]
Jul 02 13:45:45.742597 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.742557    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9eca96f3-2b55-47e3-9099-4b3b9683c02e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:45.742787 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.742624    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qtvl\" (UniqueName: \"kubernetes.io/projected/9eca96f3-2b55-47e3-9099-4b3b9683c02e-kube-api-access-6qtvl\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:45.742787 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.742663    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eca96f3-2b55-47e3-9099-4b3b9683c02e-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:45.742787 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.742693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eca96f3-2b55-47e3-9099-4b3b9683c02e-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:45.794827 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.794798    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9fqv7"
Jul 02 13:45:45.843575 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.843486    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9eca96f3-2b55-47e3-9099-4b3b9683c02e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:45.843575 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.843546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qtvl\" (UniqueName: \"kubernetes.io/projected/9eca96f3-2b55-47e3-9099-4b3b9683c02e-kube-api-access-6qtvl\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:45.843783 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.843587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eca96f3-2b55-47e3-9099-4b3b9683c02e-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:45.843783 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.843616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eca96f3-2b55-47e3-9099-4b3b9683c02e-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:45.843887 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:45.843828    2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 02 13:45:45.843937 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:45:45.843895    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eca96f3-2b55-47e3-9099-4b3b9683c02e-prometheus-operator-tls podName:9eca96f3-2b55-47e3-9099-4b3b9683c02e nodeName:}" failed. No retries permitted until 2026-07-02 13:45:46.34387459 +0000 UTC m=+83.539498095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/9eca96f3-2b55-47e3-9099-4b3b9683c02e-prometheus-operator-tls") pod "prometheus-operator-7b7d896497-b8npm" (UID: "9eca96f3-2b55-47e3-9099-4b3b9683c02e") : secret "prometheus-operator-tls" not found
Jul 02 13:45:45.844793 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.844747    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eca96f3-2b55-47e3-9099-4b3b9683c02e-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:45.846461 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.846434    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9eca96f3-2b55-47e3-9099-4b3b9683c02e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:45.852566 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:45.852542    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qtvl\" (UniqueName: \"kubernetes.io/projected/9eca96f3-2b55-47e3-9099-4b3b9683c02e-kube-api-access-6qtvl\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:46.346945 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:46.346910    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eca96f3-2b55-47e3-9099-4b3b9683c02e-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:46.349311 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:46.349293    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eca96f3-2b55-47e3-9099-4b3b9683c02e-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-b8npm\" (UID: \"9eca96f3-2b55-47e3-9099-4b3b9683c02e\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:46.489900 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:46.489855    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm"
Jul 02 13:45:46.614551 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:46.614480    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-b8npm"]
Jul 02 13:45:46.616922 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:46.616889    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eca96f3_2b55_47e3_9099_4b3b9683c02e.slice/crio-c50f329062d3787f43dd40fd7cdb126c901d29ed6f235e39f5c3be4f2998b822 WatchSource:0}: Error finding container c50f329062d3787f43dd40fd7cdb126c901d29ed6f235e39f5c3be4f2998b822: Status 404 returned error can't find the container with id c50f329062d3787f43dd40fd7cdb126c901d29ed6f235e39f5c3be4f2998b822
Jul 02 13:45:46.841709 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:46.841673    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8mlp" event={"ID":"7e76678b-b568-49d1-8c10-78d92083fbd9","Type":"ContainerStarted","Data":"9aba9e53f66b0a737eb61f225bd950ee4c868738d4642e565613e0ab484bcbd9"}
Jul 02 13:45:46.842758 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:46.842734    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm" event={"ID":"9eca96f3-2b55-47e3-9099-4b3b9683c02e","Type":"ContainerStarted","Data":"c50f329062d3787f43dd40fd7cdb126c901d29ed6f235e39f5c3be4f2998b822"}
Jul 02 13:45:47.434772 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:47.434735    2574 patch_prober.go:28] interesting pod/image-registry-ffd7db6d5-swxmp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 02 13:45:47.434939 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:47.434796    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp" podUID="cd439b40-ef2a-416e-8475-8c84e660989e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:45:47.848917 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:47.848821    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8mlp" event={"ID":"7e76678b-b568-49d1-8c10-78d92083fbd9","Type":"ContainerStarted","Data":"620ee188b7eb5224a0087c015b08627166b9891ee11fbe756203048cc95be6cc"}
Jul 02 13:45:48.855366 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:48.855333    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm" event={"ID":"9eca96f3-2b55-47e3-9099-4b3b9683c02e","Type":"ContainerStarted","Data":"9ab7bb26bf471a1277604d8f29faa7f32d1a33ca283c84da6ffeefd9c732177e"}
Jul 02 13:45:48.855830 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:48.855394    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm" event={"ID":"9eca96f3-2b55-47e3-9099-4b3b9683c02e","Type":"ContainerStarted","Data":"1e4b7fdb4b9e594dc99de2298a989da22ad0ab5df5f6060b67921ec5d07926d9"}
Jul 02 13:45:48.875536 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:48.875485    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-b8npm" podStartSLOduration=2.426942941 podStartE2EDuration="3.875466801s" totalImagesPullingTime="1.44852386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:48.875449509 +0000 UTC m=+86.071073017" watchObservedRunningTime="2026-07-02 13:45:48.875466801 +0000 UTC m=+86.071090311"
Jul 02 13:45:49.664199 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.664156    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d589669f8-m48gb"]
Jul 02 13:45:49.666314 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.666297    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.670194 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.670170    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 13:45:49.670292 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.670265    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 13:45:49.670741 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.670727    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 13:45:49.671404 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.671384    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lr5m4\""
Jul 02 13:45:49.671487 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.671436    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 13:45:49.672412 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.672277    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 13:45:49.672412 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.672320    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 13:45:49.672561 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.672442    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 13:45:49.677106 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.677088    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 13:45:49.677934 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.677915    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d589669f8-m48gb"]
Jul 02 13:45:49.776372 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.776293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-serving-cert\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.776372 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.776326    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-oauth-serving-cert\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.776372 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.776360    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-console-config\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.776547 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.776423    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rmvg\" (UniqueName: \"kubernetes.io/projected/630cdfec-1641-4b65-9163-4a368d74e657-kube-api-access-6rmvg\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.776547 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.776473    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-trusted-ca-bundle\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.776547 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.776524    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-oauth-config\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.776657 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.776554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-service-ca\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.860980 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.860939    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8mlp" event={"ID":"7e76678b-b568-49d1-8c10-78d92083fbd9","Type":"ContainerStarted","Data":"a16b105050f66dd544fbdd3ccc84ab0dd28ba9536e700e2ead042f41001a9ba7"}
Jul 02 13:45:49.876881 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.876855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rmvg\" (UniqueName: \"kubernetes.io/projected/630cdfec-1641-4b65-9163-4a368d74e657-kube-api-access-6rmvg\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.876881 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.876883    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-trusted-ca-bundle\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.877092 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.876917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-oauth-config\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.877092 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.876943    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-service-ca\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.877092 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.876985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-serving-cert\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.877092 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.877002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-oauth-serving-cert\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.877092 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.877023    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-console-config\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.877765 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.877736    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-service-ca\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.877883 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.877837    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-trusted-ca-bundle\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.877883 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.877858    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-oauth-serving-cert\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.877967 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.877858    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-console-config\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.879874 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.879849    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-oauth-config\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.880188 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.880147    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-serving-cert\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.880665 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.880620    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n8mlp" podStartSLOduration=33.810412368 podStartE2EDuration="40.880604186s" totalImagesPullingTime="7.070191818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:49.880413992 +0000 UTC m=+87.076037500" watchObservedRunningTime="2026-07-02 13:45:49.880604186 +0000 UTC m=+87.076227694"
Jul 02 13:45:49.886387 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.886362    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rmvg\" (UniqueName: \"kubernetes.io/projected/630cdfec-1641-4b65-9163-4a368d74e657-kube-api-access-6rmvg\") pod \"console-7d589669f8-m48gb\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") " pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:49.976261 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:49.976221    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:45:50.111739 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:50.111684    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d589669f8-m48gb"]
Jul 02 13:45:50.115991 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:50.115960    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630cdfec_1641_4b65_9163_4a368d74e657.slice/crio-537f90bde2a3fafe23dc83c1d96164d1eae6865dcb81dd3191a541583c317825 WatchSource:0}: Error finding container 537f90bde2a3fafe23dc83c1d96164d1eae6865dcb81dd3191a541583c317825: Status 404 returned error can't find the container with id 537f90bde2a3fafe23dc83c1d96164d1eae6865dcb81dd3191a541583c317825
Jul 02 13:45:50.865985 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:50.865952    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d589669f8-m48gb" event={"ID":"630cdfec-1641-4b65-9163-4a368d74e657","Type":"ContainerStarted","Data":"537f90bde2a3fafe23dc83c1d96164d1eae6865dcb81dd3191a541583c317825"}
Jul 02 13:45:50.989273 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:50.989242    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tjnxc"]
Jul 02 13:45:50.991835 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:50.991812    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:50.996193 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:50.995888    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 13:45:50.996193 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:50.996060    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 13:45:50.996193 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:50.996094    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9mx2c\""
Jul 02 13:45:50.997211 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:50.997187    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 13:45:51.089089 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.089056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vpl8\" (UniqueName: \"kubernetes.io/projected/dbbc86cc-3ebd-46fa-826e-04eb377d5643-kube-api-access-7vpl8\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.089089 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.089094    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.089354 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.089116    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-accelerators-collector-config\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.089354 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.089171    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-tls\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.089354 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.089202    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-wtmp\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.089354 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.089229    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbbc86cc-3ebd-46fa-826e-04eb377d5643-sys\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.089354 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.089282    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbbc86cc-3ebd-46fa-826e-04eb377d5643-metrics-client-ca\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.089602 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.089369    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-textfile\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.089602 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.089415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dbbc86cc-3ebd-46fa-826e-04eb377d5643-root\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.190679 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.190640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-textfile\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.190881 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.190704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dbbc86cc-3ebd-46fa-826e-04eb377d5643-root\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.190881 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.190742    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vpl8\" (UniqueName: \"kubernetes.io/projected/dbbc86cc-3ebd-46fa-826e-04eb377d5643-kube-api-access-7vpl8\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.190881 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.190800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.190881 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.190829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-accelerators-collector-config\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.190881 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.190836    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dbbc86cc-3ebd-46fa-826e-04eb377d5643-root\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.190881 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.190870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-tls\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.191197 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.190897    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-wtmp\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.191197 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.190923    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbbc86cc-3ebd-46fa-826e-04eb377d5643-sys\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.191197 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.190948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbbc86cc-3ebd-46fa-826e-04eb377d5643-metrics-client-ca\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.191197 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.191048    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-textfile\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.191418 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.191375    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbbc86cc-3ebd-46fa-826e-04eb377d5643-sys\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.191418 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.191404    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-wtmp\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.191589 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.191506    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbbc86cc-3ebd-46fa-826e-04eb377d5643-metrics-client-ca\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.191807 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.191778    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-accelerators-collector-config\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.193525 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.193503    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.193766 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.193751    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dbbc86cc-3ebd-46fa-826e-04eb377d5643-node-exporter-tls\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.203407 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.203381    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vpl8\" (UniqueName: \"kubernetes.io/projected/dbbc86cc-3ebd-46fa-826e-04eb377d5643-kube-api-access-7vpl8\") pod \"node-exporter-tjnxc\" (UID: \"dbbc86cc-3ebd-46fa-826e-04eb377d5643\") " pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.303899 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.303868    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tjnxc"
Jul 02 13:45:51.314744 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:51.314710    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbc86cc_3ebd_46fa_826e_04eb377d5643.slice/crio-26dd02df921780ffdca39df43f0bb071f4f69d49e27afb7f25d0f3d5119b7457 WatchSource:0}: Error finding container 26dd02df921780ffdca39df43f0bb071f4f69d49e27afb7f25d0f3d5119b7457: Status 404 returned error can't find the container with id 26dd02df921780ffdca39df43f0bb071f4f69d49e27afb7f25d0f3d5119b7457
Jul 02 13:45:51.871556 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:51.871518    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tjnxc" event={"ID":"dbbc86cc-3ebd-46fa-826e-04eb377d5643","Type":"ContainerStarted","Data":"26dd02df921780ffdca39df43f0bb071f4f69d49e27afb7f25d0f3d5119b7457"}
Jul 02 13:45:53.880816 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:53.880783    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d589669f8-m48gb" event={"ID":"630cdfec-1641-4b65-9163-4a368d74e657","Type":"ContainerStarted","Data":"766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5"}
Jul 02 13:45:53.882476 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:53.882448    2574 generic.go:356] "Generic (PLEG): container finished" podID="dbbc86cc-3ebd-46fa-826e-04eb377d5643" containerID="8cab8645af7f558aba70db281c4991d1e42624f68e050892602c4eb91c077c4c" exitCode=0
Jul 02 13:45:53.882601 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:53.882485    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tjnxc" event={"ID":"dbbc86cc-3ebd-46fa-826e-04eb377d5643","Type":"ContainerDied","Data":"8cab8645af7f558aba70db281c4991d1e42624f68e050892602c4eb91c077c4c"}
Jul 02 13:45:53.901643 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:53.901600    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d589669f8-m48gb" podStartSLOduration=1.61721691 podStartE2EDuration="4.901585516s" totalImagesPullingTime="3.284368606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:53.90029517 +0000 UTC m=+91.095918677" watchObservedRunningTime="2026-07-02 13:45:53.901585516 +0000 UTC m=+91.097209024"
Jul 02 13:45:54.892108 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:54.892075    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tjnxc" event={"ID":"dbbc86cc-3ebd-46fa-826e-04eb377d5643","Type":"ContainerStarted","Data":"7370f0c172905348156b62e4a519463ea50e47d847e69fd85d8576e305583719"}
Jul 02 13:45:54.892514 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:54.892180    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tjnxc" event={"ID":"dbbc86cc-3ebd-46fa-826e-04eb377d5643","Type":"ContainerStarted","Data":"2ef1b5bc61dd988907696759536b95408f3974ed3761992ac275ef3abcbc67ef"}
Jul 02 13:45:54.913991 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:54.913944    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tjnxc" podStartSLOduration=2.758034193 podStartE2EDuration="4.913929592s" totalImagesPullingTime="2.083548757s" totalInitContainerRuntime="72.346642ms" isStatefulPod=true podCreationTimestamp="2026-07-02 13:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:54.911818645 +0000 UTC m=+92.107442168" watchObservedRunningTime="2026-07-02 13:45:54.913929592 +0000 UTC m=+92.109553099"
Jul 02 13:45:56.068240 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.068205    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d589669f8-m48gb"]
Jul 02 13:45:56.102027 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.102001    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59f5799d74-c8klb"]
Jul 02 13:45:56.105387 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.105372    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.118885 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.118863    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f5799d74-c8klb"]
Jul 02 13:45:56.134251 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.134229    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-trusted-ca-bundle\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.134338 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.134256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-oauth-serving-cert\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.134338 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.134298    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-config\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.134338 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.134328    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-oauth-config\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.134448 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.134346    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-service-ca\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.134448 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.134362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-serving-cert\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.134448 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.134413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmp7v\" (UniqueName: \"kubernetes.io/projected/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-kube-api-access-bmp7v\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.235355 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.235306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-config\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.235530 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.235380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-oauth-config\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.235530 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.235411    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-service-ca\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.235530 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.235440    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-serving-cert\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.235530 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.235479    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmp7v\" (UniqueName: \"kubernetes.io/projected/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-kube-api-access-bmp7v\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.235530 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.235513    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-trusted-ca-bundle\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.235796 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.235536    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-oauth-serving-cert\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.236401 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.236378    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-oauth-serving-cert\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.236643 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.236617    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-service-ca\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.236797 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.236775    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-config\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.236970 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.236944    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-trusted-ca-bundle\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.238505 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.238477    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-oauth-config\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.238505 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.238493    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-serving-cert\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.259548 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.259522    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmp7v\" (UniqueName: \"kubernetes.io/projected/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-kube-api-access-bmp7v\") pod \"console-59f5799d74-c8klb\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") " pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.414124 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.414095    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:45:56.539526 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.539500    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f5799d74-c8klb"]
Jul 02 13:45:56.541689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:45:56.541661    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad548da1_4f47_40f1_a4ad_2d58f535d9f0.slice/crio-61d33e7f70ddc4fa6111b8299b9adad987352bdff05c50828bb8220da56ef3dc WatchSource:0}: Error finding container 61d33e7f70ddc4fa6111b8299b9adad987352bdff05c50828bb8220da56ef3dc: Status 404 returned error can't find the container with id 61d33e7f70ddc4fa6111b8299b9adad987352bdff05c50828bb8220da56ef3dc
Jul 02 13:45:56.902165 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.902056    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f5799d74-c8klb" event={"ID":"ad548da1-4f47-40f1-a4ad-2d58f535d9f0","Type":"ContainerStarted","Data":"6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5"}
Jul 02 13:45:56.902165 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.902104    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f5799d74-c8klb" event={"ID":"ad548da1-4f47-40f1-a4ad-2d58f535d9f0","Type":"ContainerStarted","Data":"61d33e7f70ddc4fa6111b8299b9adad987352bdff05c50828bb8220da56ef3dc"}
Jul 02 13:45:56.921879 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:56.921833    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59f5799d74-c8klb" podStartSLOduration=0.921819641 podStartE2EDuration="921.819641ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:56.92119008 +0000 UTC m=+94.116813600" watchObservedRunningTime="2026-07-02 13:45:56.921819641 +0000 UTC m=+94.117443147"
Jul 02 13:45:57.434112 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:57.434088    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:45:59.976793 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:45:59.976758    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:46:00.814467 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:00.814432    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59f6b9978d-hmh9t"
Jul 02 13:46:01.556112 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:01.556080    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f5799d74-c8klb"]
Jul 02 13:46:02.448372 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.448324    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp" podUID="cd439b40-ef2a-416e-8475-8c84e660989e" containerName="registry" containerID="cri-o://67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231" gracePeriod=30
Jul 02 13:46:02.709011 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.708953    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:46:02.792159 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.792109    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bch26\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-kube-api-access-bch26\") pod \"cd439b40-ef2a-416e-8475-8c84e660989e\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") "
Jul 02 13:46:02.792310 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.792168    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-bound-sa-token\") pod \"cd439b40-ef2a-416e-8475-8c84e660989e\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") "
Jul 02 13:46:02.792310 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.792200    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-registry-certificates\") pod \"cd439b40-ef2a-416e-8475-8c84e660989e\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") "
Jul 02 13:46:02.792310 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.792233    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-image-registry-private-configuration\") pod \"cd439b40-ef2a-416e-8475-8c84e660989e\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") "
Jul 02 13:46:02.792310 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.792273    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-trusted-ca\") pod \"cd439b40-ef2a-416e-8475-8c84e660989e\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") "
Jul 02 13:46:02.792510 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.792312    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd439b40-ef2a-416e-8475-8c84e660989e-ca-trust-extracted\") pod \"cd439b40-ef2a-416e-8475-8c84e660989e\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") "
Jul 02 13:46:02.792510 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.792388    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls\") pod \"cd439b40-ef2a-416e-8475-8c84e660989e\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") "
Jul 02 13:46:02.792510 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.792421    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-installation-pull-secrets\") pod \"cd439b40-ef2a-416e-8475-8c84e660989e\" (UID: \"cd439b40-ef2a-416e-8475-8c84e660989e\") "
Jul 02 13:46:02.793088 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.792738    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cd439b40-ef2a-416e-8475-8c84e660989e" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:46:02.793088 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.793045    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cd439b40-ef2a-416e-8475-8c84e660989e" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:46:02.794954 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.794925    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cd439b40-ef2a-416e-8475-8c84e660989e" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:46:02.794954 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.794950    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cd439b40-ef2a-416e-8475-8c84e660989e" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:46:02.795126 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.795032    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-kube-api-access-bch26" (OuterVolumeSpecName: "kube-api-access-bch26") pod "cd439b40-ef2a-416e-8475-8c84e660989e" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e"). InnerVolumeSpecName "kube-api-access-bch26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:46:02.795126 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.795051    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cd439b40-ef2a-416e-8475-8c84e660989e" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:46:02.795126 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.795101    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cd439b40-ef2a-416e-8475-8c84e660989e" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:46:02.800407 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.800385    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd439b40-ef2a-416e-8475-8c84e660989e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cd439b40-ef2a-416e-8475-8c84e660989e" (UID: "cd439b40-ef2a-416e-8475-8c84e660989e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:46:02.893375 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.893321    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-image-registry-private-configuration\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:02.893375 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.893369    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-trusted-ca\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:02.893375 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.893384    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd439b40-ef2a-416e-8475-8c84e660989e-ca-trust-extracted\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:02.893627 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.893396    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-registry-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:02.893627 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.893407    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd439b40-ef2a-416e-8475-8c84e660989e-installation-pull-secrets\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:02.893627 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.893420    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bch26\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-kube-api-access-bch26\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:02.893627 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.893431    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd439b40-ef2a-416e-8475-8c84e660989e-bound-sa-token\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:02.893627 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.893442    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd439b40-ef2a-416e-8475-8c84e660989e-registry-certificates\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:02.929237 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.929201    2574 generic.go:356] "Generic (PLEG): container finished" podID="cd439b40-ef2a-416e-8475-8c84e660989e" containerID="67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231" exitCode=0
Jul 02 13:46:02.929414 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.929251    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp" event={"ID":"cd439b40-ef2a-416e-8475-8c84e660989e","Type":"ContainerDied","Data":"67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231"}
Jul 02 13:46:02.929414 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.929270    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp"
Jul 02 13:46:02.929414 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.929309    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ffd7db6d5-swxmp" event={"ID":"cd439b40-ef2a-416e-8475-8c84e660989e","Type":"ContainerDied","Data":"7bc2a7f4f28291f5ade6d082e950b2c8d7f289fa651c7599415c52f7efaaa86d"}
Jul 02 13:46:02.929414 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.929329    2574 scope.go:117] "RemoveContainer" containerID="67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231"
Jul 02 13:46:02.938122 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.938105    2574 scope.go:117] "RemoveContainer" containerID="67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231"
Jul 02 13:46:02.938446 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:46:02.938427    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231\": container with ID starting with 67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231 not found: ID does not exist" containerID="67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231"
Jul 02 13:46:02.938487 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.938454    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231"} err="failed to get container status \"67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231\": rpc error: code = NotFound desc = could not find container \"67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231\": container with ID starting with 67f98216609581981956013e32aef18a4a4e10d9b00a46e05593ffab13127231 not found: ID does not exist"
Jul 02 13:46:02.950336 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.950317    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-ffd7db6d5-swxmp"]
Jul 02 13:46:02.954018 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:02.953997    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-ffd7db6d5-swxmp"]
Jul 02 13:46:03.349615 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:03.349585    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd439b40-ef2a-416e-8475-8c84e660989e" path="/var/lib/kubelet/pods/cd439b40-ef2a-416e-8475-8c84e660989e/volumes"
Jul 02 13:46:06.415113 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:06.415082    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:46:06.794875 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:06.794794    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tlx9j"
Jul 02 13:46:21.087949 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.087886    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7d589669f8-m48gb" podUID="630cdfec-1641-4b65-9163-4a368d74e657" containerName="console" containerID="cri-o://766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5" gracePeriod=15
Jul 02 13:46:21.332884 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.332856    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:46:21.452743 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.452717    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-service-ca\") pod \"630cdfec-1641-4b65-9163-4a368d74e657\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") "
Jul 02 13:46:21.452900 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.452751    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-console-config\") pod \"630cdfec-1641-4b65-9163-4a368d74e657\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") "
Jul 02 13:46:21.452900 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.452789    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-oauth-serving-cert\") pod \"630cdfec-1641-4b65-9163-4a368d74e657\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") "
Jul 02 13:46:21.453008 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.452990    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-oauth-config\") pod \"630cdfec-1641-4b65-9163-4a368d74e657\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") "
Jul 02 13:46:21.453049 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.453033    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rmvg\" (UniqueName: \"kubernetes.io/projected/630cdfec-1641-4b65-9163-4a368d74e657-kube-api-access-6rmvg\") pod \"630cdfec-1641-4b65-9163-4a368d74e657\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") "
Jul 02 13:46:21.453107 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.453085    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-serving-cert\") pod \"630cdfec-1641-4b65-9163-4a368d74e657\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") "
Jul 02 13:46:21.453256 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.453161    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-trusted-ca-bundle\") pod \"630cdfec-1641-4b65-9163-4a368d74e657\" (UID: \"630cdfec-1641-4b65-9163-4a368d74e657\") "
Jul 02 13:46:21.453256 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.453162    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-console-config" (OuterVolumeSpecName: "console-config") pod "630cdfec-1641-4b65-9163-4a368d74e657" (UID: "630cdfec-1641-4b65-9163-4a368d74e657"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:46:21.453256 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.453198    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "630cdfec-1641-4b65-9163-4a368d74e657" (UID: "630cdfec-1641-4b65-9163-4a368d74e657"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:46:21.453256 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.453211    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-service-ca" (OuterVolumeSpecName: "service-ca") pod "630cdfec-1641-4b65-9163-4a368d74e657" (UID: "630cdfec-1641-4b65-9163-4a368d74e657"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:46:21.453432 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.453361    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-service-ca\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:21.453432 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.453378    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-console-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:21.453432 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.453391    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-oauth-serving-cert\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:21.453628 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.453606    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "630cdfec-1641-4b65-9163-4a368d74e657" (UID: "630cdfec-1641-4b65-9163-4a368d74e657"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:46:21.455349 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.455310    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "630cdfec-1641-4b65-9163-4a368d74e657" (UID: "630cdfec-1641-4b65-9163-4a368d74e657"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:46:21.455424 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.455378    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "630cdfec-1641-4b65-9163-4a368d74e657" (UID: "630cdfec-1641-4b65-9163-4a368d74e657"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:46:21.455424 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.455392    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630cdfec-1641-4b65-9163-4a368d74e657-kube-api-access-6rmvg" (OuterVolumeSpecName: "kube-api-access-6rmvg") pod "630cdfec-1641-4b65-9163-4a368d74e657" (UID: "630cdfec-1641-4b65-9163-4a368d74e657"). InnerVolumeSpecName "kube-api-access-6rmvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:46:21.553947 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.553908    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-oauth-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:21.553947 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.553939    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rmvg\" (UniqueName: \"kubernetes.io/projected/630cdfec-1641-4b65-9163-4a368d74e657-kube-api-access-6rmvg\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:21.553947 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.553949    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/630cdfec-1641-4b65-9163-4a368d74e657-console-serving-cert\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:21.553947 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:21.553957    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/630cdfec-1641-4b65-9163-4a368d74e657-trusted-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:22.002541 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:22.002511    2574 generic.go:356] "Generic (PLEG): container finished" podID="630cdfec-1641-4b65-9163-4a368d74e657" containerID="766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5" exitCode=0
Jul 02 13:46:22.002709 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:22.002566    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d589669f8-m48gb"
Jul 02 13:46:22.002709 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:22.002599    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d589669f8-m48gb" event={"ID":"630cdfec-1641-4b65-9163-4a368d74e657","Type":"ContainerDied","Data":"766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5"}
Jul 02 13:46:22.002709 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:22.002649    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d589669f8-m48gb" event={"ID":"630cdfec-1641-4b65-9163-4a368d74e657","Type":"ContainerDied","Data":"537f90bde2a3fafe23dc83c1d96164d1eae6865dcb81dd3191a541583c317825"}
Jul 02 13:46:22.002709 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:22.002668    2574 scope.go:117] "RemoveContainer" containerID="766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5"
Jul 02 13:46:22.011251 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:22.011230    2574 scope.go:117] "RemoveContainer" containerID="766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5"
Jul 02 13:46:22.011509 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:46:22.011490    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5\": container with ID starting with 766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5 not found: ID does not exist" containerID="766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5"
Jul 02 13:46:22.011564 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:22.011517    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5"} err="failed to get container status \"766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5\": rpc error: code = NotFound desc = could not find container \"766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5\": container with ID starting with 766a7ae1804077b8d0ea8c86660d4df53862b08ed81e7fbdceb15a6b155413c5 not found: ID does not exist"
Jul 02 13:46:22.024759 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:22.024736    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d589669f8-m48gb"]
Jul 02 13:46:22.028562 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:22.028544    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d589669f8-m48gb"]
Jul 02 13:46:23.349855 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:23.349823    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="630cdfec-1641-4b65-9163-4a368d74e657" path="/var/lib/kubelet/pods/630cdfec-1641-4b65-9163-4a368d74e657/volumes"
Jul 02 13:46:25.015001 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:25.014961    2574 generic.go:356] "Generic (PLEG): container finished" podID="b236de76-d2f6-469a-8bed-f59597794af0" containerID="1a426b27b0219dbb209d2e0068938278d5d927112f2ac7f082d35895da643796" exitCode=0
Jul 02 13:46:25.015379 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:25.015032    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-s6bjn" event={"ID":"b236de76-d2f6-469a-8bed-f59597794af0","Type":"ContainerDied","Data":"1a426b27b0219dbb209d2e0068938278d5d927112f2ac7f082d35895da643796"}
Jul 02 13:46:25.015425 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:25.015404    2574 scope.go:117] "RemoveContainer" containerID="1a426b27b0219dbb209d2e0068938278d5d927112f2ac7f082d35895da643796"
Jul 02 13:46:26.020668 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.020634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-s6bjn" event={"ID":"b236de76-d2f6-469a-8bed-f59597794af0","Type":"ContainerStarted","Data":"02505446fefb39f836b7b34e2d24dcb9d1a3f633754e070bdf89f40002a8a170"}
Jul 02 13:46:26.576996 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.576955    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-59f5799d74-c8klb" podUID="ad548da1-4f47-40f1-a4ad-2d58f535d9f0" containerName="console" containerID="cri-o://6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5" gracePeriod=15
Jul 02 13:46:26.830765 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.830704    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:46:26.997642 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.997601    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-config\") pod \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") "
Jul 02 13:46:26.997642 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.997647    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-serving-cert\") pod \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") "
Jul 02 13:46:26.997883 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.997693    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-service-ca\") pod \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") "
Jul 02 13:46:26.997883 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.997798    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmp7v\" (UniqueName: \"kubernetes.io/projected/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-kube-api-access-bmp7v\") pod \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") "
Jul 02 13:46:26.997883 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.997844    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-trusted-ca-bundle\") pod \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") "
Jul 02 13:46:26.997883 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.997869    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-oauth-serving-cert\") pod \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") "
Jul 02 13:46:26.998094 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.997912    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-oauth-config\") pod \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\" (UID: \"ad548da1-4f47-40f1-a4ad-2d58f535d9f0\") "
Jul 02 13:46:26.998094 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.997983    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-config" (OuterVolumeSpecName: "console-config") pod "ad548da1-4f47-40f1-a4ad-2d58f535d9f0" (UID: "ad548da1-4f47-40f1-a4ad-2d58f535d9f0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:46:26.998230 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.998162    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:26.998230 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.998188    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-service-ca" (OuterVolumeSpecName: "service-ca") pod "ad548da1-4f47-40f1-a4ad-2d58f535d9f0" (UID: "ad548da1-4f47-40f1-a4ad-2d58f535d9f0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:46:26.998337 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.998317    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ad548da1-4f47-40f1-a4ad-2d58f535d9f0" (UID: "ad548da1-4f47-40f1-a4ad-2d58f535d9f0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:46:26.998385 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.998321    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ad548da1-4f47-40f1-a4ad-2d58f535d9f0" (UID: "ad548da1-4f47-40f1-a4ad-2d58f535d9f0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:46:26.999988 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.999957    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ad548da1-4f47-40f1-a4ad-2d58f535d9f0" (UID: "ad548da1-4f47-40f1-a4ad-2d58f535d9f0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:46:27.000429 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:26.999989    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ad548da1-4f47-40f1-a4ad-2d58f535d9f0" (UID: "ad548da1-4f47-40f1-a4ad-2d58f535d9f0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:46:27.000429 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.000043    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-kube-api-access-bmp7v" (OuterVolumeSpecName: "kube-api-access-bmp7v") pod "ad548da1-4f47-40f1-a4ad-2d58f535d9f0" (UID: "ad548da1-4f47-40f1-a4ad-2d58f535d9f0"). InnerVolumeSpecName "kube-api-access-bmp7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:46:27.025195 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.025166    2574 generic.go:356] "Generic (PLEG): container finished" podID="ad548da1-4f47-40f1-a4ad-2d58f535d9f0" containerID="6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5" exitCode=0
Jul 02 13:46:27.025522 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.025218    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f5799d74-c8klb"
Jul 02 13:46:27.025522 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.025221    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f5799d74-c8klb" event={"ID":"ad548da1-4f47-40f1-a4ad-2d58f535d9f0","Type":"ContainerDied","Data":"6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5"}
Jul 02 13:46:27.025522 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.025270    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f5799d74-c8klb" event={"ID":"ad548da1-4f47-40f1-a4ad-2d58f535d9f0","Type":"ContainerDied","Data":"61d33e7f70ddc4fa6111b8299b9adad987352bdff05c50828bb8220da56ef3dc"}
Jul 02 13:46:27.025522 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.025296    2574 scope.go:117] "RemoveContainer" containerID="6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5"
Jul 02 13:46:27.033727 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.033710    2574 scope.go:117] "RemoveContainer" containerID="6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5"
Jul 02 13:46:27.034028 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:46:27.034003    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5\": container with ID starting with 6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5 not found: ID does not exist" containerID="6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5"
Jul 02 13:46:27.034100 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.034041    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5"} err="failed to get container status \"6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5\": rpc error: code = NotFound desc = could not find container \"6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5\": container with ID starting with 6011ef695ab12328fc6c3a41840aa4e40bfb3d6b24e0e21f00970a7fd382c0a5 not found: ID does not exist"
Jul 02 13:46:27.046998 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.046976    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f5799d74-c8klb"]
Jul 02 13:46:27.050933 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.050912    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59f5799d74-c8klb"]
Jul 02 13:46:27.098822 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.098770    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-service-ca\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:27.098822 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.098789    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmp7v\" (UniqueName: \"kubernetes.io/projected/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-kube-api-access-bmp7v\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:27.098822 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.098799    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-trusted-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:27.098822 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.098809    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-oauth-serving-cert\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:27.098822 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.098817    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-oauth-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:27.099017 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.098826    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad548da1-4f47-40f1-a4ad-2d58f535d9f0-console-serving-cert\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:46:27.353215 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:27.350951    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad548da1-4f47-40f1-a4ad-2d58f535d9f0" path="/var/lib/kubelet/pods/ad548da1-4f47-40f1-a4ad-2d58f535d9f0/volumes"
Jul 02 13:46:30.038807 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:30.038776    2574 generic.go:356] "Generic (PLEG): container finished" podID="bd56974d-3878-413a-9cb8-b56a9db313fc" containerID="c0c784d7440f5ccdfbefd975c31bdad6b8f10881d12ed0736ec732728deab027" exitCode=0
Jul 02 13:46:30.039209 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:30.038815    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf" event={"ID":"bd56974d-3878-413a-9cb8-b56a9db313fc","Type":"ContainerDied","Data":"c0c784d7440f5ccdfbefd975c31bdad6b8f10881d12ed0736ec732728deab027"}
Jul 02 13:46:30.039209 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:30.039092    2574 scope.go:117] "RemoveContainer" containerID="c0c784d7440f5ccdfbefd975c31bdad6b8f10881d12ed0736ec732728deab027"
Jul 02 13:46:31.044387 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:46:31.044355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-djtvf" event={"ID":"bd56974d-3878-413a-9cb8-b56a9db313fc","Type":"ContainerStarted","Data":"e7f732d53e06a273f580a4ac0fa30731c18ffa21cd2dd2635c194026ffd57c91"}
Jul 02 13:47:09.757616 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.757587    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68f87c8ddb-fz9jk"]
Jul 02 13:47:09.758262 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.757894    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad548da1-4f47-40f1-a4ad-2d58f535d9f0" containerName="console"
Jul 02 13:47:09.758262 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.757909    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad548da1-4f47-40f1-a4ad-2d58f535d9f0" containerName="console"
Jul 02 13:47:09.758262 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.757948    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630cdfec-1641-4b65-9163-4a368d74e657" containerName="console"
Jul 02 13:47:09.758262 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.757956    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="630cdfec-1641-4b65-9163-4a368d74e657" containerName="console"
Jul 02 13:47:09.758262 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.757969    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd439b40-ef2a-416e-8475-8c84e660989e" containerName="registry"
Jul 02 13:47:09.758262 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.757977    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd439b40-ef2a-416e-8475-8c84e660989e" containerName="registry"
Jul 02 13:47:09.758262 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.758047    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="630cdfec-1641-4b65-9163-4a368d74e657" containerName="console"
Jul 02 13:47:09.758262 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.758059    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad548da1-4f47-40f1-a4ad-2d58f535d9f0" containerName="console"
Jul 02 13:47:09.758262 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.758070    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd439b40-ef2a-416e-8475-8c84e660989e" containerName="registry"
Jul 02 13:47:09.760627 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.760586    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.763850 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.763828    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 13:47:09.763988 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.763899    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 13:47:09.764126 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.764107    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 13:47:09.765303 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.765283    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 13:47:09.765395 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.765316    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 13:47:09.765395 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.765283    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 13:47:09.765946 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.765922    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lr5m4\""
Jul 02 13:47:09.766212 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.765950    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 13:47:09.778948 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.778925    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f87c8ddb-fz9jk"]
Jul 02 13:47:09.779417 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.779399    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 13:47:09.804411 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.804390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-oauth-serving-cert\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.804513 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.804433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdmtq\" (UniqueName: \"kubernetes.io/projected/6178871a-a66c-4a02-9382-94eae0fc8ccb-kube-api-access-rdmtq\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.804513 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.804453    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-trusted-ca-bundle\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.804513 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.804477    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-config\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.804513 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.804501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-service-ca\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.804684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.804526    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-serving-cert\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.804684 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.804543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-oauth-config\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.905796 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.905755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdmtq\" (UniqueName: \"kubernetes.io/projected/6178871a-a66c-4a02-9382-94eae0fc8ccb-kube-api-access-rdmtq\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.905796 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.905796    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-trusted-ca-bundle\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.906045 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.905817    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-config\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.906045 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.905833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-service-ca\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.906045 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.905865    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-serving-cert\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.906045 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.905994    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-oauth-config\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.906274 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.906074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-oauth-serving-cert\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.906620 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.906595    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-service-ca\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.906726 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.906668    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-config\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.906787 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.906735    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-oauth-serving-cert\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.906905 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.906792    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-trusted-ca-bundle\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.908495 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.908471    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-oauth-config\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.908590 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.908471    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-serving-cert\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:09.915710 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:09.915692    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdmtq\" (UniqueName: \"kubernetes.io/projected/6178871a-a66c-4a02-9382-94eae0fc8ccb-kube-api-access-rdmtq\") pod \"console-68f87c8ddb-fz9jk\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") " pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:10.079086 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:10.079002    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:10.211611 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:10.211590    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f87c8ddb-fz9jk"]
Jul 02 13:47:10.213887 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:47:10.213857    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6178871a_a66c_4a02_9382_94eae0fc8ccb.slice/crio-14a24f38d4f1fbfdbb23eb3d3b34d10a40b2f79ac0b35379d9007aca0e6dfe55 WatchSource:0}: Error finding container 14a24f38d4f1fbfdbb23eb3d3b34d10a40b2f79ac0b35379d9007aca0e6dfe55: Status 404 returned error can't find the container with id 14a24f38d4f1fbfdbb23eb3d3b34d10a40b2f79ac0b35379d9007aca0e6dfe55
Jul 02 13:47:11.180072 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:11.180037    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f87c8ddb-fz9jk" event={"ID":"6178871a-a66c-4a02-9382-94eae0fc8ccb","Type":"ContainerStarted","Data":"e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7"}
Jul 02 13:47:11.180461 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:11.180094    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f87c8ddb-fz9jk" event={"ID":"6178871a-a66c-4a02-9382-94eae0fc8ccb","Type":"ContainerStarted","Data":"14a24f38d4f1fbfdbb23eb3d3b34d10a40b2f79ac0b35379d9007aca0e6dfe55"}
Jul 02 13:47:11.202723 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:11.202669    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68f87c8ddb-fz9jk" podStartSLOduration=2.20265339 podStartE2EDuration="2.20265339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:47:11.202048696 +0000 UTC m=+168.397672203" watchObservedRunningTime="2026-07-02 13:47:11.20265339 +0000 UTC m=+168.398276896"
Jul 02 13:47:20.079599 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:20.079554    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:20.080008 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:20.079623    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:20.084377 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:20.084356    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:47:20.213619 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:47:20.213595    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:49:01.145098 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.145015    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"]
Jul 02 13:49:01.148482 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.148461    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:01.151205 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.151180    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lnjvp\""
Jul 02 13:49:01.151306 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.151286    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 02 13:49:01.152210 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.152193    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 02 13:49:01.163193 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.163176    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"]
Jul 02 13:49:01.261472 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.261444    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:01.261640 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.261504    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:01.261640 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.261594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr7jv\" (UniqueName: \"kubernetes.io/projected/4429e18b-a52c-450a-84de-5152719dc0e7-kube-api-access-vr7jv\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:01.362193 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.362158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:01.362355 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.362222    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7jv\" (UniqueName: \"kubernetes.io/projected/4429e18b-a52c-450a-84de-5152719dc0e7-kube-api-access-vr7jv\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:01.362355 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.362260    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:01.362603 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.362582    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:01.362603 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.362597    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:01.371504 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.371474    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr7jv\" (UniqueName: \"kubernetes.io/projected/4429e18b-a52c-450a-84de-5152719dc0e7-kube-api-access-vr7jv\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:01.457463 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.457434    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:01.587414 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:01.587387    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"]
Jul 02 13:49:01.590927 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:49:01.590895    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4429e18b_a52c_450a_84de_5152719dc0e7.slice/crio-9d14502eaa069456bc0a4639e7dda2e76bf75ef9caf62abee00380844f1ba56d WatchSource:0}: Error finding container 9d14502eaa069456bc0a4639e7dda2e76bf75ef9caf62abee00380844f1ba56d: Status 404 returned error can't find the container with id 9d14502eaa069456bc0a4639e7dda2e76bf75ef9caf62abee00380844f1ba56d
Jul 02 13:49:02.562609 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:02.562569    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf" event={"ID":"4429e18b-a52c-450a-84de-5152719dc0e7","Type":"ContainerStarted","Data":"9d14502eaa069456bc0a4639e7dda2e76bf75ef9caf62abee00380844f1ba56d"}
Jul 02 13:49:08.585500 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:08.585464    2574 generic.go:356] "Generic (PLEG): container finished" podID="4429e18b-a52c-450a-84de-5152719dc0e7" containerID="607fe4dc014211997a8b09fe615bee2bb5536a4f57f884d08764793ca0e8349a" exitCode=0
Jul 02 13:49:08.585880 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:08.585538    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf" event={"ID":"4429e18b-a52c-450a-84de-5152719dc0e7","Type":"ContainerDied","Data":"607fe4dc014211997a8b09fe615bee2bb5536a4f57f884d08764793ca0e8349a"}
Jul 02 13:49:11.598340 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:11.598255    2574 generic.go:356] "Generic (PLEG): container finished" podID="4429e18b-a52c-450a-84de-5152719dc0e7" containerID="5f9903aaf6f468f57775ad0aecade4dea8aa6c26d5868baa936cd12526bf241b" exitCode=0
Jul 02 13:49:11.598340 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:11.598300    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf" event={"ID":"4429e18b-a52c-450a-84de-5152719dc0e7","Type":"ContainerDied","Data":"5f9903aaf6f468f57775ad0aecade4dea8aa6c26d5868baa936cd12526bf241b"}
Jul 02 13:49:18.626515 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:18.626482    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf" event={"ID":"4429e18b-a52c-450a-84de-5152719dc0e7","Type":"ContainerStarted","Data":"150fb5c67a667c2f3fb1cdb99f9e047e746c6d302ddc2d6d4c48a9f8aae6bf27"}
Jul 02 13:49:18.651897 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:18.651854    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf" podStartSLOduration=1.9738590440000001 podStartE2EDuration="17.651837218s" totalImagesPullingTime="15.677978174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:49:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:49:18.650394693 +0000 UTC m=+295.846018201" watchObservedRunningTime="2026-07-02 13:49:18.651837218 +0000 UTC m=+295.847460724"
Jul 02 13:49:19.632128 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:19.632091    2574 generic.go:356] "Generic (PLEG): container finished" podID="4429e18b-a52c-450a-84de-5152719dc0e7" containerID="150fb5c67a667c2f3fb1cdb99f9e047e746c6d302ddc2d6d4c48a9f8aae6bf27" exitCode=0
Jul 02 13:49:19.632536 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:19.632182    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf" event={"ID":"4429e18b-a52c-450a-84de-5152719dc0e7","Type":"ContainerDied","Data":"150fb5c67a667c2f3fb1cdb99f9e047e746c6d302ddc2d6d4c48a9f8aae6bf27"}
Jul 02 13:49:20.757839 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:20.757816    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:20.813560 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:20.813531    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-util\") pod \"4429e18b-a52c-450a-84de-5152719dc0e7\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") "
Jul 02 13:49:20.813739 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:20.813622    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-bundle\") pod \"4429e18b-a52c-450a-84de-5152719dc0e7\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") "
Jul 02 13:49:20.813739 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:20.813674    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr7jv\" (UniqueName: \"kubernetes.io/projected/4429e18b-a52c-450a-84de-5152719dc0e7-kube-api-access-vr7jv\") pod \"4429e18b-a52c-450a-84de-5152719dc0e7\" (UID: \"4429e18b-a52c-450a-84de-5152719dc0e7\") "
Jul 02 13:49:20.814370 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:20.814341    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-bundle" (OuterVolumeSpecName: "bundle") pod "4429e18b-a52c-450a-84de-5152719dc0e7" (UID: "4429e18b-a52c-450a-84de-5152719dc0e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:49:20.815818 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:20.815792    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4429e18b-a52c-450a-84de-5152719dc0e7-kube-api-access-vr7jv" (OuterVolumeSpecName: "kube-api-access-vr7jv") pod "4429e18b-a52c-450a-84de-5152719dc0e7" (UID: "4429e18b-a52c-450a-84de-5152719dc0e7"). InnerVolumeSpecName "kube-api-access-vr7jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:49:20.818917 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:20.818899    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-util" (OuterVolumeSpecName: "util") pod "4429e18b-a52c-450a-84de-5152719dc0e7" (UID: "4429e18b-a52c-450a-84de-5152719dc0e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:49:20.914401 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:20.914381    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:49:20.914401 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:20.914401    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr7jv\" (UniqueName: \"kubernetes.io/projected/4429e18b-a52c-450a-84de-5152719dc0e7-kube-api-access-vr7jv\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:49:20.914519 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:20.914410    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4429e18b-a52c-450a-84de-5152719dc0e7-util\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:49:21.640679 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:21.640603    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf"
Jul 02 13:49:21.640819 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:21.640602    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589nfmnf" event={"ID":"4429e18b-a52c-450a-84de-5152719dc0e7","Type":"ContainerDied","Data":"9d14502eaa069456bc0a4639e7dda2e76bf75ef9caf62abee00380844f1ba56d"}
Jul 02 13:49:21.640819 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:21.640723    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d14502eaa069456bc0a4639e7dda2e76bf75ef9caf62abee00380844f1ba56d"
Jul 02 13:49:23.273993 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:23.273964    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 13:49:23.278521 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:23.278501    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 13:49:23.280186 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:23.280159    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 13:49:23.283539 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:23.283521    2574 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 13:49:23.284796 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:23.284777    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 13:49:24.137154 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.137109    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"]
Jul 02 13:49:24.139222 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.137362    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4429e18b-a52c-450a-84de-5152719dc0e7" containerName="extract"
Jul 02 13:49:24.139222 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.137372    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4429e18b-a52c-450a-84de-5152719dc0e7" containerName="extract"
Jul 02 13:49:24.139222 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.137391    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4429e18b-a52c-450a-84de-5152719dc0e7" containerName="util"
Jul 02 13:49:24.139222 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.137396    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4429e18b-a52c-450a-84de-5152719dc0e7" containerName="util"
Jul 02 13:49:24.139222 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.137408    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4429e18b-a52c-450a-84de-5152719dc0e7" containerName="pull"
Jul 02 13:49:24.139222 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.137413    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4429e18b-a52c-450a-84de-5152719dc0e7" containerName="pull"
Jul 02 13:49:24.139222 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.137453    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4429e18b-a52c-450a-84de-5152719dc0e7" containerName="extract"
Jul 02 13:49:24.140049 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.140034    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"
Jul 02 13:49:24.144566 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.144542    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 02 13:49:24.144678 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.144571    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 02 13:49:24.144678 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.144616    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 02 13:49:24.144678 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.144638    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-p6wmb\""
Jul 02 13:49:24.156393 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.156374    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"]
Jul 02 13:49:24.237294 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.237265    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/40168914-29d6-4b6f-a1e1-a4fee768a90f-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9\" (UID: \"40168914-29d6-4b6f-a1e1-a4fee768a90f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"
Jul 02 13:49:24.237479 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.237306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phgzn\" (UniqueName: \"kubernetes.io/projected/40168914-29d6-4b6f-a1e1-a4fee768a90f-kube-api-access-phgzn\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9\" (UID: \"40168914-29d6-4b6f-a1e1-a4fee768a90f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"
Jul 02 13:49:24.337833 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.337787    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/40168914-29d6-4b6f-a1e1-a4fee768a90f-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9\" (UID: \"40168914-29d6-4b6f-a1e1-a4fee768a90f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"
Jul 02 13:49:24.338220 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.337855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phgzn\" (UniqueName: \"kubernetes.io/projected/40168914-29d6-4b6f-a1e1-a4fee768a90f-kube-api-access-phgzn\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9\" (UID: \"40168914-29d6-4b6f-a1e1-a4fee768a90f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"
Jul 02 13:49:24.340033 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.340016    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/40168914-29d6-4b6f-a1e1-a4fee768a90f-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9\" (UID: \"40168914-29d6-4b6f-a1e1-a4fee768a90f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"
Jul 02 13:49:24.352657 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.352633    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phgzn\" (UniqueName: \"kubernetes.io/projected/40168914-29d6-4b6f-a1e1-a4fee768a90f-kube-api-access-phgzn\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9\" (UID: \"40168914-29d6-4b6f-a1e1-a4fee768a90f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"
Jul 02 13:49:24.450486 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.450460    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"
Jul 02 13:49:24.578022 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.577973    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"]
Jul 02 13:49:24.580166 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:49:24.580130    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40168914_29d6_4b6f_a1e1_a4fee768a90f.slice/crio-cf5f9f67525520c98a7ca5bec386d0fff5f629752c52cc855cef567860a48546 WatchSource:0}: Error finding container cf5f9f67525520c98a7ca5bec386d0fff5f629752c52cc855cef567860a48546: Status 404 returned error can't find the container with id cf5f9f67525520c98a7ca5bec386d0fff5f629752c52cc855cef567860a48546
Jul 02 13:49:24.582002 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.581983    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:49:24.652794 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:24.652760    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9" event={"ID":"40168914-29d6-4b6f-a1e1-a4fee768a90f","Type":"ContainerStarted","Data":"cf5f9f67525520c98a7ca5bec386d0fff5f629752c52cc855cef567860a48546"}
Jul 02 13:49:33.692504 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:33.692469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9" event={"ID":"40168914-29d6-4b6f-a1e1-a4fee768a90f","Type":"ContainerStarted","Data":"888a65375f6ae8eff65bd6632384f36c330db21649aa91e74dc16ec06019b235"}
Jul 02 13:49:33.692894 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:33.692603    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"
Jul 02 13:49:33.717188 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:33.717127    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9" podStartSLOduration=1.125186515 podStartE2EDuration="9.717110414s" totalImagesPullingTime="8.591923899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:49:33.716054623 +0000 UTC m=+310.911678131" watchObservedRunningTime="2026-07-02 13:49:33.717110414 +0000 UTC m=+310.912733923"
Jul 02 13:49:33.942426 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:33.942393    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-pxftw"]
Jul 02 13:49:33.945530 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:33.945489    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:33.948182 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:33.948157    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 02 13:49:33.948284 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:33.948214    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-4fp42\""
Jul 02 13:49:33.948284 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:33.948239    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 02 13:49:33.957171 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:33.957147    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-pxftw"]
Jul 02 13:49:34.013386 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.013356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ee221aa-54c2-49d6-a705-fca7edf19184-certificates\") pod \"keda-operator-68bff59c-pxftw\" (UID: \"5ee221aa-54c2-49d6-a705-fca7edf19184\") " pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:34.013548 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.013394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5ee221aa-54c2-49d6-a705-fca7edf19184-cabundle0\") pod \"keda-operator-68bff59c-pxftw\" (UID: \"5ee221aa-54c2-49d6-a705-fca7edf19184\") " pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:34.013548 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.013422    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9bf\" (UniqueName: \"kubernetes.io/projected/5ee221aa-54c2-49d6-a705-fca7edf19184-kube-api-access-8b9bf\") pod \"keda-operator-68bff59c-pxftw\" (UID: \"5ee221aa-54c2-49d6-a705-fca7edf19184\") " pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:34.114791 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.114762    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ee221aa-54c2-49d6-a705-fca7edf19184-certificates\") pod \"keda-operator-68bff59c-pxftw\" (UID: \"5ee221aa-54c2-49d6-a705-fca7edf19184\") " pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:34.114791 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.114797    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5ee221aa-54c2-49d6-a705-fca7edf19184-cabundle0\") pod \"keda-operator-68bff59c-pxftw\" (UID: \"5ee221aa-54c2-49d6-a705-fca7edf19184\") " pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:34.114975 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.114835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9bf\" (UniqueName: \"kubernetes.io/projected/5ee221aa-54c2-49d6-a705-fca7edf19184-kube-api-access-8b9bf\") pod \"keda-operator-68bff59c-pxftw\" (UID: \"5ee221aa-54c2-49d6-a705-fca7edf19184\") " pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:34.114975 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:49:34.114908    2574 secret.go:281] references non-existent secret key: ca.crt
Jul 02 13:49:34.114975 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:49:34.114925    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 02 13:49:34.114975 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:49:34.114937    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-pxftw: references non-existent secret key: ca.crt
Jul 02 13:49:34.115099 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:49:34.114993    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ee221aa-54c2-49d6-a705-fca7edf19184-certificates podName:5ee221aa-54c2-49d6-a705-fca7edf19184 nodeName:}" failed. No retries permitted until 2026-07-02 13:49:34.614974677 +0000 UTC m=+311.810598164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5ee221aa-54c2-49d6-a705-fca7edf19184-certificates") pod "keda-operator-68bff59c-pxftw" (UID: "5ee221aa-54c2-49d6-a705-fca7edf19184") : references non-existent secret key: ca.crt
Jul 02 13:49:34.115444 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.115425    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5ee221aa-54c2-49d6-a705-fca7edf19184-cabundle0\") pod \"keda-operator-68bff59c-pxftw\" (UID: \"5ee221aa-54c2-49d6-a705-fca7edf19184\") " pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:34.124946 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.124922    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9bf\" (UniqueName: \"kubernetes.io/projected/5ee221aa-54c2-49d6-a705-fca7edf19184-kube-api-access-8b9bf\") pod \"keda-operator-68bff59c-pxftw\" (UID: \"5ee221aa-54c2-49d6-a705-fca7edf19184\") " pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:34.291308 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.291242    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"]
Jul 02 13:49:34.294460 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.294447    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:34.296934 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.296916    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 02 13:49:34.306341 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.306318    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"]
Jul 02 13:49:34.315937 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.315915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-49rv4\" (UID: \"71b4c05c-bd03-46d3-8ace-1ac7142ba03d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:34.316033 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.315959    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-certificates\") pod \"keda-metrics-apiserver-559f75f947-49rv4\" (UID: \"71b4c05c-bd03-46d3-8ace-1ac7142ba03d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:34.316033 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.315985    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvbf2\" (UniqueName: \"kubernetes.io/projected/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-kube-api-access-gvbf2\") pod \"keda-metrics-apiserver-559f75f947-49rv4\" (UID: \"71b4c05c-bd03-46d3-8ace-1ac7142ba03d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:34.416938 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.416906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvbf2\" (UniqueName: \"kubernetes.io/projected/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-kube-api-access-gvbf2\") pod \"keda-metrics-apiserver-559f75f947-49rv4\" (UID: \"71b4c05c-bd03-46d3-8ace-1ac7142ba03d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:34.417081 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.416965    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-49rv4\" (UID: \"71b4c05c-bd03-46d3-8ace-1ac7142ba03d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:34.417081 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.417009    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-certificates\") pod \"keda-metrics-apiserver-559f75f947-49rv4\" (UID: \"71b4c05c-bd03-46d3-8ace-1ac7142ba03d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:34.417196 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:49:34.417093    2574 secret.go:281] references non-existent secret key: tls.crt
Jul 02 13:49:34.417196 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:49:34.417104    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 02 13:49:34.417196 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:49:34.417121    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-49rv4: references non-existent secret key: tls.crt
Jul 02 13:49:34.417196 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:49:34.417184    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-certificates podName:71b4c05c-bd03-46d3-8ace-1ac7142ba03d nodeName:}" failed. No retries permitted until 2026-07-02 13:49:34.917170625 +0000 UTC m=+312.112794110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-certificates") pod "keda-metrics-apiserver-559f75f947-49rv4" (UID: "71b4c05c-bd03-46d3-8ace-1ac7142ba03d") : references non-existent secret key: tls.crt
Jul 02 13:49:34.417394 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.417377    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-49rv4\" (UID: \"71b4c05c-bd03-46d3-8ace-1ac7142ba03d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:34.425484 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.425462    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvbf2\" (UniqueName: \"kubernetes.io/projected/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-kube-api-access-gvbf2\") pod \"keda-metrics-apiserver-559f75f947-49rv4\" (UID: \"71b4c05c-bd03-46d3-8ace-1ac7142ba03d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:34.619103 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.619016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ee221aa-54c2-49d6-a705-fca7edf19184-certificates\") pod \"keda-operator-68bff59c-pxftw\" (UID: \"5ee221aa-54c2-49d6-a705-fca7edf19184\") " pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:34.621348 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.621321    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5ee221aa-54c2-49d6-a705-fca7edf19184-certificates\") pod \"keda-operator-68bff59c-pxftw\" (UID: \"5ee221aa-54c2-49d6-a705-fca7edf19184\") " pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:34.683545 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.683513    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-2kckr"]
Jul 02 13:49:34.686639 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.686621    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-2kckr"
Jul 02 13:49:34.688894 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.688870    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 02 13:49:34.696306 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.696285    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-2kckr"]
Jul 02 13:49:34.719597 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.719572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40a4d38a-0d54-4a52-bdeb-b3b58581a59c-certificates\") pod \"keda-admission-5fbd5c4644-2kckr\" (UID: \"40a4d38a-0d54-4a52-bdeb-b3b58581a59c\") " pod="openshift-keda/keda-admission-5fbd5c4644-2kckr"
Jul 02 13:49:34.719738 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.719682    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w99sf\" (UniqueName: \"kubernetes.io/projected/40a4d38a-0d54-4a52-bdeb-b3b58581a59c-kube-api-access-w99sf\") pod \"keda-admission-5fbd5c4644-2kckr\" (UID: \"40a4d38a-0d54-4a52-bdeb-b3b58581a59c\") " pod="openshift-keda/keda-admission-5fbd5c4644-2kckr"
Jul 02 13:49:34.820603 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.820565    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w99sf\" (UniqueName: \"kubernetes.io/projected/40a4d38a-0d54-4a52-bdeb-b3b58581a59c-kube-api-access-w99sf\") pod \"keda-admission-5fbd5c4644-2kckr\" (UID: \"40a4d38a-0d54-4a52-bdeb-b3b58581a59c\") " pod="openshift-keda/keda-admission-5fbd5c4644-2kckr"
Jul 02 13:49:34.820785 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.820644    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40a4d38a-0d54-4a52-bdeb-b3b58581a59c-certificates\") pod \"keda-admission-5fbd5c4644-2kckr\" (UID: \"40a4d38a-0d54-4a52-bdeb-b3b58581a59c\") " pod="openshift-keda/keda-admission-5fbd5c4644-2kckr"
Jul 02 13:49:34.820785 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:49:34.820777    2574 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 02 13:49:34.820906 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:49:34.820804    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-2kckr: secret "keda-admission-webhooks-certs" not found
Jul 02 13:49:34.820906 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:49:34.820860    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40a4d38a-0d54-4a52-bdeb-b3b58581a59c-certificates podName:40a4d38a-0d54-4a52-bdeb-b3b58581a59c nodeName:}" failed. No retries permitted until 2026-07-02 13:49:35.320845639 +0000 UTC m=+312.516469137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/40a4d38a-0d54-4a52-bdeb-b3b58581a59c-certificates") pod "keda-admission-5fbd5c4644-2kckr" (UID: "40a4d38a-0d54-4a52-bdeb-b3b58581a59c") : secret "keda-admission-webhooks-certs" not found
Jul 02 13:49:34.829414 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.829384    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w99sf\" (UniqueName: \"kubernetes.io/projected/40a4d38a-0d54-4a52-bdeb-b3b58581a59c-kube-api-access-w99sf\") pod \"keda-admission-5fbd5c4644-2kckr\" (UID: \"40a4d38a-0d54-4a52-bdeb-b3b58581a59c\") " pod="openshift-keda/keda-admission-5fbd5c4644-2kckr"
Jul 02 13:49:34.856074 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.856049    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:34.921201 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.921167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-certificates\") pod \"keda-metrics-apiserver-559f75f947-49rv4\" (UID: \"71b4c05c-bd03-46d3-8ace-1ac7142ba03d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:34.924389 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:34.924358    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/71b4c05c-bd03-46d3-8ace-1ac7142ba03d-certificates\") pod \"keda-metrics-apiserver-559f75f947-49rv4\" (UID: \"71b4c05c-bd03-46d3-8ace-1ac7142ba03d\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:35.008120 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:35.007773    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-pxftw"]
Jul 02 13:49:35.010178 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:49:35.010129    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee221aa_54c2_49d6_a705_fca7edf19184.slice/crio-4f1d881661f8fdee14c1bfa50ed6534fffad931827736ba43af928a44bb8a043 WatchSource:0}: Error finding container 4f1d881661f8fdee14c1bfa50ed6534fffad931827736ba43af928a44bb8a043: Status 404 returned error can't find the container with id 4f1d881661f8fdee14c1bfa50ed6534fffad931827736ba43af928a44bb8a043
Jul 02 13:49:35.204292 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:35.204249    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:35.325386 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:35.325351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40a4d38a-0d54-4a52-bdeb-b3b58581a59c-certificates\") pod \"keda-admission-5fbd5c4644-2kckr\" (UID: \"40a4d38a-0d54-4a52-bdeb-b3b58581a59c\") " pod="openshift-keda/keda-admission-5fbd5c4644-2kckr"
Jul 02 13:49:35.327849 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:35.327827    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40a4d38a-0d54-4a52-bdeb-b3b58581a59c-certificates\") pod \"keda-admission-5fbd5c4644-2kckr\" (UID: \"40a4d38a-0d54-4a52-bdeb-b3b58581a59c\") " pod="openshift-keda/keda-admission-5fbd5c4644-2kckr"
Jul 02 13:49:35.338156 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:35.338062    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"]
Jul 02 13:49:35.340638 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:49:35.340613    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b4c05c_bd03_46d3_8ace_1ac7142ba03d.slice/crio-3955ac2804f983df9f56f81c09ff83aa8154005e555dd2d267b2cf269ab6bc78 WatchSource:0}: Error finding container 3955ac2804f983df9f56f81c09ff83aa8154005e555dd2d267b2cf269ab6bc78: Status 404 returned error can't find the container with id 3955ac2804f983df9f56f81c09ff83aa8154005e555dd2d267b2cf269ab6bc78
Jul 02 13:49:35.598265 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:35.598194    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-2kckr"
Jul 02 13:49:35.703597 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:35.703557    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-pxftw" event={"ID":"5ee221aa-54c2-49d6-a705-fca7edf19184","Type":"ContainerStarted","Data":"4f1d881661f8fdee14c1bfa50ed6534fffad931827736ba43af928a44bb8a043"}
Jul 02 13:49:35.705504 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:35.705477    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4" event={"ID":"71b4c05c-bd03-46d3-8ace-1ac7142ba03d","Type":"ContainerStarted","Data":"3955ac2804f983df9f56f81c09ff83aa8154005e555dd2d267b2cf269ab6bc78"}
Jul 02 13:49:35.731018 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:35.730983    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-2kckr"]
Jul 02 13:49:35.733772 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:49:35.733748    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a4d38a_0d54_4a52_bdeb_b3b58581a59c.slice/crio-38132edffed897c9bd523ed857a70cf620abc9b345b5c418fd8f3c726109553e WatchSource:0}: Error finding container 38132edffed897c9bd523ed857a70cf620abc9b345b5c418fd8f3c726109553e: Status 404 returned error can't find the container with id 38132edffed897c9bd523ed857a70cf620abc9b345b5c418fd8f3c726109553e
Jul 02 13:49:36.713576 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:36.713535    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-2kckr" event={"ID":"40a4d38a-0d54-4a52-bdeb-b3b58581a59c","Type":"ContainerStarted","Data":"38132edffed897c9bd523ed857a70cf620abc9b345b5c418fd8f3c726109553e"}
Jul 02 13:49:39.727940 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:39.727843    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4" event={"ID":"71b4c05c-bd03-46d3-8ace-1ac7142ba03d","Type":"ContainerStarted","Data":"4d4814752d2f9ddf3a740847f97147bff9a12eeade1721aee24ec8e15842101b"}
Jul 02 13:49:39.728405 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:39.727933    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:39.729397 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:39.729366    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-pxftw" event={"ID":"5ee221aa-54c2-49d6-a705-fca7edf19184","Type":"ContainerStarted","Data":"011b33e23434e6e227e49ea4c62cdd1b87584d347e07a7a8ab74d8cdb924fbe1"}
Jul 02 13:49:39.729517 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:39.729427    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:49:39.730881 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:39.730860    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-2kckr" event={"ID":"40a4d38a-0d54-4a52-bdeb-b3b58581a59c","Type":"ContainerStarted","Data":"dff282dffa101fcecd125dfaf194d966724bb9e92a3cedf99007eae5e0bc41db"}
Jul 02 13:49:39.731014 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:39.730972    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-2kckr"
Jul 02 13:49:39.748594 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:39.748553    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4" podStartSLOduration=1.506561434 podStartE2EDuration="5.748540649s" totalImagesPullingTime="4.241979215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:49:39.746576696 +0000 UTC m=+316.942200204" watchObservedRunningTime="2026-07-02 13:49:39.748540649 +0000 UTC m=+316.944164156"
Jul 02 13:49:39.763787 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:39.763735    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-pxftw" podStartSLOduration=2.186825455 podStartE2EDuration="6.763718179s" totalImagesPullingTime="4.576892724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:49:39.762918209 +0000 UTC m=+316.958541717" watchObservedRunningTime="2026-07-02 13:49:39.763718179 +0000 UTC m=+316.959341688"
Jul 02 13:49:39.781707 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:39.781654    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-2kckr" podStartSLOduration=1.938001627 podStartE2EDuration="5.781639229s" totalImagesPullingTime="3.843637602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:49:39.781089477 +0000 UTC m=+316.976712984" watchObservedRunningTime="2026-07-02 13:49:39.781639229 +0000 UTC m=+316.977262735"
Jul 02 13:49:50.740026 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:50.740002    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-49rv4"
Jul 02 13:49:54.699286 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:49:54.699260    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-tjfn9"
Jul 02 13:50:00.737854 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:00.737824    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-pxftw"
Jul 02 13:50:00.738261 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:00.737904    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-2kckr"
Jul 02 13:50:42.833218 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.833183    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7c747fbc6-jtvgt"]
Jul 02 13:50:42.835494 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.835478    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:50:42.838520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.838494    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 02 13:50:42.838520 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.838514    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 13:50:42.839685 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.839666    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-zm25j\""
Jul 02 13:50:42.839685 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.839680    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 13:50:42.850615 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.850595    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7c747fbc6-jtvgt"]
Jul 02 13:50:42.856542 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.856514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw94d\" (UniqueName: \"kubernetes.io/projected/31110817-ab43-4945-94ce-7cc36a54d1c8-kube-api-access-zw94d\") pod \"kserve-controller-manager-7c747fbc6-jtvgt\" (UID: \"31110817-ab43-4945-94ce-7cc36a54d1c8\") " pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:50:42.856644 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.856608    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31110817-ab43-4945-94ce-7cc36a54d1c8-cert\") pod \"kserve-controller-manager-7c747fbc6-jtvgt\" (UID: \"31110817-ab43-4945-94ce-7cc36a54d1c8\") " pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:50:42.870879 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.870857    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-zslbv"]
Jul 02 13:50:42.872929 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.872915    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zslbv"
Jul 02 13:50:42.875590 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.875574    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 13:50:42.875692 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.875622    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7wn6m\""
Jul 02 13:50:42.884696 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.884669    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zslbv"]
Jul 02 13:50:42.957551 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.957516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31110817-ab43-4945-94ce-7cc36a54d1c8-cert\") pod \"kserve-controller-manager-7c747fbc6-jtvgt\" (UID: \"31110817-ab43-4945-94ce-7cc36a54d1c8\") " pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:50:42.957726 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.957576    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f9ac7535-9962-43af-9a3d-e79a0ff67849-data\") pod \"seaweedfs-64568bcd49-zslbv\" (UID: \"f9ac7535-9962-43af-9a3d-e79a0ff67849\") " pod="kserve/seaweedfs-64568bcd49-zslbv"
Jul 02 13:50:42.957726 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.957600    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw94d\" (UniqueName: \"kubernetes.io/projected/31110817-ab43-4945-94ce-7cc36a54d1c8-kube-api-access-zw94d\") pod \"kserve-controller-manager-7c747fbc6-jtvgt\" (UID: \"31110817-ab43-4945-94ce-7cc36a54d1c8\") " pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:50:42.957726 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.957621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds5v9\" (UniqueName: \"kubernetes.io/projected/f9ac7535-9962-43af-9a3d-e79a0ff67849-kube-api-access-ds5v9\") pod \"seaweedfs-64568bcd49-zslbv\" (UID: \"f9ac7535-9962-43af-9a3d-e79a0ff67849\") " pod="kserve/seaweedfs-64568bcd49-zslbv"
Jul 02 13:50:42.957726 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:50:42.957673    2574 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 02 13:50:42.957862 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:50:42.957752    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31110817-ab43-4945-94ce-7cc36a54d1c8-cert podName:31110817-ab43-4945-94ce-7cc36a54d1c8 nodeName:}" failed. No retries permitted until 2026-07-02 13:50:43.457733166 +0000 UTC m=+380.653356651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/31110817-ab43-4945-94ce-7cc36a54d1c8-cert") pod "kserve-controller-manager-7c747fbc6-jtvgt" (UID: "31110817-ab43-4945-94ce-7cc36a54d1c8") : secret "kserve-webhook-server-cert" not found
Jul 02 13:50:42.968278 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:42.968250    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw94d\" (UniqueName: \"kubernetes.io/projected/31110817-ab43-4945-94ce-7cc36a54d1c8-kube-api-access-zw94d\") pod \"kserve-controller-manager-7c747fbc6-jtvgt\" (UID: \"31110817-ab43-4945-94ce-7cc36a54d1c8\") " pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:50:43.058898 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.058863    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds5v9\" (UniqueName: \"kubernetes.io/projected/f9ac7535-9962-43af-9a3d-e79a0ff67849-kube-api-access-ds5v9\") pod \"seaweedfs-64568bcd49-zslbv\" (UID: \"f9ac7535-9962-43af-9a3d-e79a0ff67849\") " pod="kserve/seaweedfs-64568bcd49-zslbv"
Jul 02 13:50:43.059070 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.058968    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f9ac7535-9962-43af-9a3d-e79a0ff67849-data\") pod \"seaweedfs-64568bcd49-zslbv\" (UID: \"f9ac7535-9962-43af-9a3d-e79a0ff67849\") " pod="kserve/seaweedfs-64568bcd49-zslbv"
Jul 02 13:50:43.059383 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.059367    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f9ac7535-9962-43af-9a3d-e79a0ff67849-data\") pod \"seaweedfs-64568bcd49-zslbv\" (UID: \"f9ac7535-9962-43af-9a3d-e79a0ff67849\") " pod="kserve/seaweedfs-64568bcd49-zslbv"
Jul 02 13:50:43.069494 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.069473    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds5v9\" (UniqueName: \"kubernetes.io/projected/f9ac7535-9962-43af-9a3d-e79a0ff67849-kube-api-access-ds5v9\") pod \"seaweedfs-64568bcd49-zslbv\" (UID: \"f9ac7535-9962-43af-9a3d-e79a0ff67849\") " pod="kserve/seaweedfs-64568bcd49-zslbv"
Jul 02 13:50:43.181129 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.181092    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zslbv"
Jul 02 13:50:43.307757 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.307616    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zslbv"]
Jul 02 13:50:43.310321 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:50:43.310294    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ac7535_9962_43af_9a3d_e79a0ff67849.slice/crio-6aa70fbc87d19e6689813e2b378d1799a6c2e53274c6780384ce1ab516d2b54f WatchSource:0}: Error finding container 6aa70fbc87d19e6689813e2b378d1799a6c2e53274c6780384ce1ab516d2b54f: Status 404 returned error can't find the container with id 6aa70fbc87d19e6689813e2b378d1799a6c2e53274c6780384ce1ab516d2b54f
Jul 02 13:50:43.462296 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.462211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31110817-ab43-4945-94ce-7cc36a54d1c8-cert\") pod \"kserve-controller-manager-7c747fbc6-jtvgt\" (UID: \"31110817-ab43-4945-94ce-7cc36a54d1c8\") " pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:50:43.464585 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.464557    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31110817-ab43-4945-94ce-7cc36a54d1c8-cert\") pod \"kserve-controller-manager-7c747fbc6-jtvgt\" (UID: \"31110817-ab43-4945-94ce-7cc36a54d1c8\") " pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:50:43.745464 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.745390    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:50:43.878688 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.878653    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7c747fbc6-jtvgt"]
Jul 02 13:50:43.882498 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:50:43.882464    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31110817_ab43_4945_94ce_7cc36a54d1c8.slice/crio-720e8f3d93e4b881f427cf36d4078fed7f75663d90a06b81171faca0d52ca753 WatchSource:0}: Error finding container 720e8f3d93e4b881f427cf36d4078fed7f75663d90a06b81171faca0d52ca753: Status 404 returned error can't find the container with id 720e8f3d93e4b881f427cf36d4078fed7f75663d90a06b81171faca0d52ca753
Jul 02 13:50:43.996423 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.996337    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt" event={"ID":"31110817-ab43-4945-94ce-7cc36a54d1c8","Type":"ContainerStarted","Data":"720e8f3d93e4b881f427cf36d4078fed7f75663d90a06b81171faca0d52ca753"}
Jul 02 13:50:43.997995 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:43.997961    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zslbv" event={"ID":"f9ac7535-9962-43af-9a3d-e79a0ff67849","Type":"ContainerStarted","Data":"6aa70fbc87d19e6689813e2b378d1799a6c2e53274c6780384ce1ab516d2b54f"}
Jul 02 13:50:48.019128 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:48.019094    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zslbv" event={"ID":"f9ac7535-9962-43af-9a3d-e79a0ff67849","Type":"ContainerStarted","Data":"ea029dfc9201948e9015fe7e90ad9a55d7b7eda7ae4d2892ada1b0d17be249d3"}
Jul 02 13:50:48.019665 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:48.019199    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-zslbv"
Jul 02 13:50:48.042977 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:48.042933    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-zslbv" podStartSLOduration=2.39997941 podStartE2EDuration="6.042918837s" totalImagesPullingTime="3.642939427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:50:48.042709648 +0000 UTC m=+385.238333155" watchObservedRunningTime="2026-07-02 13:50:48.042918837 +0000 UTC m=+385.238542343"
Jul 02 13:50:49.024477 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:49.024441    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt" event={"ID":"31110817-ab43-4945-94ce-7cc36a54d1c8","Type":"ContainerStarted","Data":"0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861"}
Jul 02 13:50:49.024975 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:49.024529    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:50:49.050859 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:49.048129    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt" podStartSLOduration=2.517187877 podStartE2EDuration="7.048109698s" totalImagesPullingTime="4.530921821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:50:49.046675951 +0000 UTC m=+386.242299461" watchObservedRunningTime="2026-07-02 13:50:49.048109698 +0000 UTC m=+386.243733206"
Jul 02 13:50:54.026796 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:50:54.026769    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-zslbv"
Jul 02 13:51:02.302564 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.302531    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bffccbfdd-brlkm"]
Jul 02 13:51:02.306841 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.306824    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.323675 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.323644    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bffccbfdd-brlkm"]
Jul 02 13:51:02.427637 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.427605    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-console-config\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.427834 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.427652    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqhp\" (UniqueName: \"kubernetes.io/projected/0b96240a-3651-4d67-ade5-1cd9bb698e85-kube-api-access-tvqhp\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.427834 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.427675    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-trusted-ca-bundle\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.427834 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.427692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-oauth-serving-cert\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.427834 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.427722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b96240a-3651-4d67-ade5-1cd9bb698e85-console-oauth-config\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.427834 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.427740    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b96240a-3651-4d67-ade5-1cd9bb698e85-console-serving-cert\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.427834 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.427791    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-service-ca\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.528544 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.528505    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqhp\" (UniqueName: \"kubernetes.io/projected/0b96240a-3651-4d67-ade5-1cd9bb698e85-kube-api-access-tvqhp\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.528544 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.528548    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-trusted-ca-bundle\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.528731 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.528563    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-oauth-serving-cert\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.528731 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.528586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b96240a-3651-4d67-ade5-1cd9bb698e85-console-oauth-config\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.528731 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.528615    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b96240a-3651-4d67-ade5-1cd9bb698e85-console-serving-cert\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.528731 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.528647    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-service-ca\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.528731 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.528711    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-console-config\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.529469 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.529440    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-oauth-serving-cert\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.529609 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.529448    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-console-config\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.529609 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.529511    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-trusted-ca-bundle\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.529609 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.529521    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b96240a-3651-4d67-ade5-1cd9bb698e85-service-ca\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.531069 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.531041    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b96240a-3651-4d67-ade5-1cd9bb698e85-console-oauth-config\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.531196 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.531176    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b96240a-3651-4d67-ade5-1cd9bb698e85-console-serving-cert\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.539078 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.539054    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqhp\" (UniqueName: \"kubernetes.io/projected/0b96240a-3651-4d67-ade5-1cd9bb698e85-kube-api-access-tvqhp\") pod \"console-7bffccbfdd-brlkm\" (UID: \"0b96240a-3651-4d67-ade5-1cd9bb698e85\") " pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.616003 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.615919    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:02.752813 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:02.752619    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bffccbfdd-brlkm"]
Jul 02 13:51:02.755882 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:51:02.755856    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b96240a_3651_4d67_ade5_1cd9bb698e85.slice/crio-c79d4a05b2a069817138a9d39a95dda007837aa4c567b3ae993d70f07c0497fb WatchSource:0}: Error finding container c79d4a05b2a069817138a9d39a95dda007837aa4c567b3ae993d70f07c0497fb: Status 404 returned error can't find the container with id c79d4a05b2a069817138a9d39a95dda007837aa4c567b3ae993d70f07c0497fb
Jul 02 13:51:03.080977 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:03.080940    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bffccbfdd-brlkm" event={"ID":"0b96240a-3651-4d67-ade5-1cd9bb698e85","Type":"ContainerStarted","Data":"f03bac7db17a6d814f3e544734af4cdca776a72101e8bd1a93796a8583f1cd65"}
Jul 02 13:51:03.081178 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:03.080993    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bffccbfdd-brlkm" event={"ID":"0b96240a-3651-4d67-ade5-1cd9bb698e85","Type":"ContainerStarted","Data":"c79d4a05b2a069817138a9d39a95dda007837aa4c567b3ae993d70f07c0497fb"}
Jul 02 13:51:03.103499 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:03.103452    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bffccbfdd-brlkm" podStartSLOduration=1.103439238 podStartE2EDuration="1.103439238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:51:03.101336601 +0000 UTC m=+400.296960108" watchObservedRunningTime="2026-07-02 13:51:03.103439238 +0000 UTC m=+400.299062744"
Jul 02 13:51:12.616191 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:12.616128    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:12.616580 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:12.616207    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:12.620956 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:12.620935    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:13.124836 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:13.124807    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bffccbfdd-brlkm"
Jul 02 13:51:13.173814 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:13.173783    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f87c8ddb-fz9jk"]
Jul 02 13:51:19.356560 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.356530    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7c747fbc6-jtvgt"]
Jul 02 13:51:19.357071 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.356698    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt" podUID="31110817-ab43-4945-94ce-7cc36a54d1c8" containerName="manager" containerID="cri-o://0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861" gracePeriod=10
Jul 02 13:51:19.362204 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.362180    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:51:19.399018 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.398996    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7c747fbc6-wfl57"]
Jul 02 13:51:19.408972 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.408951    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c747fbc6-wfl57"
Jul 02 13:51:19.418388 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.418366    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7c747fbc6-wfl57"]
Jul 02 13:51:19.572649 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.572621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc7hl\" (UniqueName: \"kubernetes.io/projected/87e3e28f-d41a-49f1-940f-f01fc260cc30-kube-api-access-tc7hl\") pod \"kserve-controller-manager-7c747fbc6-wfl57\" (UID: \"87e3e28f-d41a-49f1-940f-f01fc260cc30\") " pod="kserve/kserve-controller-manager-7c747fbc6-wfl57"
Jul 02 13:51:19.572789 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.572665    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87e3e28f-d41a-49f1-940f-f01fc260cc30-cert\") pod \"kserve-controller-manager-7c747fbc6-wfl57\" (UID: \"87e3e28f-d41a-49f1-940f-f01fc260cc30\") " pod="kserve/kserve-controller-manager-7c747fbc6-wfl57"
Jul 02 13:51:19.609763 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.609710    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:51:19.673465 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.673436    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc7hl\" (UniqueName: \"kubernetes.io/projected/87e3e28f-d41a-49f1-940f-f01fc260cc30-kube-api-access-tc7hl\") pod \"kserve-controller-manager-7c747fbc6-wfl57\" (UID: \"87e3e28f-d41a-49f1-940f-f01fc260cc30\") " pod="kserve/kserve-controller-manager-7c747fbc6-wfl57"
Jul 02 13:51:19.673616 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.673483    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87e3e28f-d41a-49f1-940f-f01fc260cc30-cert\") pod \"kserve-controller-manager-7c747fbc6-wfl57\" (UID: \"87e3e28f-d41a-49f1-940f-f01fc260cc30\") " pod="kserve/kserve-controller-manager-7c747fbc6-wfl57"
Jul 02 13:51:19.675657 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.675636    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87e3e28f-d41a-49f1-940f-f01fc260cc30-cert\") pod \"kserve-controller-manager-7c747fbc6-wfl57\" (UID: \"87e3e28f-d41a-49f1-940f-f01fc260cc30\") " pod="kserve/kserve-controller-manager-7c747fbc6-wfl57"
Jul 02 13:51:19.683534 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.683510    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc7hl\" (UniqueName: \"kubernetes.io/projected/87e3e28f-d41a-49f1-940f-f01fc260cc30-kube-api-access-tc7hl\") pod \"kserve-controller-manager-7c747fbc6-wfl57\" (UID: \"87e3e28f-d41a-49f1-940f-f01fc260cc30\") " pod="kserve/kserve-controller-manager-7c747fbc6-wfl57"
Jul 02 13:51:19.750321 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.750296    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c747fbc6-wfl57"
Jul 02 13:51:19.774568 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.774545    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31110817-ab43-4945-94ce-7cc36a54d1c8-cert\") pod \"31110817-ab43-4945-94ce-7cc36a54d1c8\" (UID: \"31110817-ab43-4945-94ce-7cc36a54d1c8\") "
Jul 02 13:51:19.774661 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.774608    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw94d\" (UniqueName: \"kubernetes.io/projected/31110817-ab43-4945-94ce-7cc36a54d1c8-kube-api-access-zw94d\") pod \"31110817-ab43-4945-94ce-7cc36a54d1c8\" (UID: \"31110817-ab43-4945-94ce-7cc36a54d1c8\") "
Jul 02 13:51:19.776583 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.776558    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31110817-ab43-4945-94ce-7cc36a54d1c8-cert" (OuterVolumeSpecName: "cert") pod "31110817-ab43-4945-94ce-7cc36a54d1c8" (UID: "31110817-ab43-4945-94ce-7cc36a54d1c8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:51:19.776669 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.776587    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31110817-ab43-4945-94ce-7cc36a54d1c8-kube-api-access-zw94d" (OuterVolumeSpecName: "kube-api-access-zw94d") pod "31110817-ab43-4945-94ce-7cc36a54d1c8" (UID: "31110817-ab43-4945-94ce-7cc36a54d1c8"). InnerVolumeSpecName "kube-api-access-zw94d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:51:19.875842 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.875800    2574 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31110817-ab43-4945-94ce-7cc36a54d1c8-cert\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:51:19.875842 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.875836    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw94d\" (UniqueName: \"kubernetes.io/projected/31110817-ab43-4945-94ce-7cc36a54d1c8-kube-api-access-zw94d\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:51:19.910399 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:19.910374    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7c747fbc6-wfl57"]
Jul 02 13:51:19.912689 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:51:19.912667    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e3e28f_d41a_49f1_940f_f01fc260cc30.slice/crio-a2c4ed54713ae20241f20a8e53a8cdff9d1104e9efbcbcbfbd732756d81e0b9c WatchSource:0}: Error finding container a2c4ed54713ae20241f20a8e53a8cdff9d1104e9efbcbcbfbd732756d81e0b9c: Status 404 returned error can't find the container with id a2c4ed54713ae20241f20a8e53a8cdff9d1104e9efbcbcbfbd732756d81e0b9c
Jul 02 13:51:20.150708 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:20.150628    2574 generic.go:356] "Generic (PLEG): container finished" podID="31110817-ab43-4945-94ce-7cc36a54d1c8" containerID="0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861" exitCode=0
Jul 02 13:51:20.150708 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:20.150673    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt"
Jul 02 13:51:20.150903 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:20.150670    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt" event={"ID":"31110817-ab43-4945-94ce-7cc36a54d1c8","Type":"ContainerDied","Data":"0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861"}
Jul 02 13:51:20.150903 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:20.150787    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c747fbc6-jtvgt" event={"ID":"31110817-ab43-4945-94ce-7cc36a54d1c8","Type":"ContainerDied","Data":"720e8f3d93e4b881f427cf36d4078fed7f75663d90a06b81171faca0d52ca753"}
Jul 02 13:51:20.150903 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:20.150815    2574 scope.go:117] "RemoveContainer" containerID="0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861"
Jul 02 13:51:20.151968 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:20.151950    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c747fbc6-wfl57" event={"ID":"87e3e28f-d41a-49f1-940f-f01fc260cc30","Type":"ContainerStarted","Data":"a2c4ed54713ae20241f20a8e53a8cdff9d1104e9efbcbcbfbd732756d81e0b9c"}
Jul 02 13:51:20.159886 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:20.159853    2574 scope.go:117] "RemoveContainer" containerID="0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861"
Jul 02 13:51:20.160120 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:51:20.160103    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861\": container with ID starting with 0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861 not found: ID does not exist" containerID="0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861"
Jul 02 13:51:20.160260 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:20.160228    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861"} err="failed to get container status \"0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861\": rpc error: code = NotFound desc = could not find container \"0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861\": container with ID starting with 0f389c764ba9ae3e0dfaba22b252a3203296293bf684d261ed53a79c831af861 not found: ID does not exist"
Jul 02 13:51:20.173125 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:20.173105    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7c747fbc6-jtvgt"]
Jul 02 13:51:20.176705 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:20.176686    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7c747fbc6-jtvgt"]
Jul 02 13:51:21.160464 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:21.160415    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c747fbc6-wfl57" event={"ID":"87e3e28f-d41a-49f1-940f-f01fc260cc30","Type":"ContainerStarted","Data":"1073cbee971142c5f6cd4774c5332eff765c98385da8bf23c16f460c777f225c"}
Jul 02 13:51:21.160953 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:21.160923    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7c747fbc6-wfl57"
Jul 02 13:51:21.182732 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:21.182690    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7c747fbc6-wfl57" podStartSLOduration=1.599711831 podStartE2EDuration="2.182676972s" totalImagesPullingTime="582.965141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:51:21.181761001 +0000 UTC m=+418.377384509" watchObservedRunningTime="2026-07-02 13:51:21.182676972 +0000 UTC m=+418.378300480"
Jul 02 13:51:21.350936 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:21.350894    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31110817-ab43-4945-94ce-7cc36a54d1c8" path="/var/lib/kubelet/pods/31110817-ab43-4945-94ce-7cc36a54d1c8/volumes"
Jul 02 13:51:38.194346 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.194303    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-68f87c8ddb-fz9jk" podUID="6178871a-a66c-4a02-9382-94eae0fc8ccb" containerName="console" containerID="cri-o://e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7" gracePeriod=15
Jul 02 13:51:38.433310 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.433285    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:51:38.520442 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.520351    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-serving-cert\") pod \"6178871a-a66c-4a02-9382-94eae0fc8ccb\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") "
Jul 02 13:51:38.520442 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.520389    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-config\") pod \"6178871a-a66c-4a02-9382-94eae0fc8ccb\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") "
Jul 02 13:51:38.520442 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.520421    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-oauth-config\") pod \"6178871a-a66c-4a02-9382-94eae0fc8ccb\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") "
Jul 02 13:51:38.520700 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.520452    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-oauth-serving-cert\") pod \"6178871a-a66c-4a02-9382-94eae0fc8ccb\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") "
Jul 02 13:51:38.520700 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.520478    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-service-ca\") pod \"6178871a-a66c-4a02-9382-94eae0fc8ccb\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") "
Jul 02 13:51:38.520700 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.520507    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-trusted-ca-bundle\") pod \"6178871a-a66c-4a02-9382-94eae0fc8ccb\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") "
Jul 02 13:51:38.520700 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.520535    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdmtq\" (UniqueName: \"kubernetes.io/projected/6178871a-a66c-4a02-9382-94eae0fc8ccb-kube-api-access-rdmtq\") pod \"6178871a-a66c-4a02-9382-94eae0fc8ccb\" (UID: \"6178871a-a66c-4a02-9382-94eae0fc8ccb\") "
Jul 02 13:51:38.520903 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.520881    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6178871a-a66c-4a02-9382-94eae0fc8ccb" (UID: "6178871a-a66c-4a02-9382-94eae0fc8ccb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:51:38.521089 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.521061    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-service-ca" (OuterVolumeSpecName: "service-ca") pod "6178871a-a66c-4a02-9382-94eae0fc8ccb" (UID: "6178871a-a66c-4a02-9382-94eae0fc8ccb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:51:38.521189 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.521111    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6178871a-a66c-4a02-9382-94eae0fc8ccb" (UID: "6178871a-a66c-4a02-9382-94eae0fc8ccb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:51:38.521242 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.521211    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-config" (OuterVolumeSpecName: "console-config") pod "6178871a-a66c-4a02-9382-94eae0fc8ccb" (UID: "6178871a-a66c-4a02-9382-94eae0fc8ccb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:51:38.522597 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.522576    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6178871a-a66c-4a02-9382-94eae0fc8ccb" (UID: "6178871a-a66c-4a02-9382-94eae0fc8ccb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:51:38.523005 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.522985    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6178871a-a66c-4a02-9382-94eae0fc8ccb" (UID: "6178871a-a66c-4a02-9382-94eae0fc8ccb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:51:38.523005 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.522996    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6178871a-a66c-4a02-9382-94eae0fc8ccb-kube-api-access-rdmtq" (OuterVolumeSpecName: "kube-api-access-rdmtq") pod "6178871a-a66c-4a02-9382-94eae0fc8ccb" (UID: "6178871a-a66c-4a02-9382-94eae0fc8ccb"). InnerVolumeSpecName "kube-api-access-rdmtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:51:38.621408 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.621380    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-oauth-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:51:38.621408 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.621402    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-oauth-serving-cert\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:51:38.621408 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.621412    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-service-ca\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:51:38.621587 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.621432    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-trusted-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:51:38.621587 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.621441    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdmtq\" (UniqueName: \"kubernetes.io/projected/6178871a-a66c-4a02-9382-94eae0fc8ccb-kube-api-access-rdmtq\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:51:38.621587 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.621450    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-serving-cert\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:51:38.621587 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:38.621458    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6178871a-a66c-4a02-9382-94eae0fc8ccb-console-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:51:39.243181 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:39.243149    2574 generic.go:356] "Generic (PLEG): container finished" podID="6178871a-a66c-4a02-9382-94eae0fc8ccb" containerID="e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7" exitCode=0
Jul 02 13:51:39.243609 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:39.243191    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f87c8ddb-fz9jk"
Jul 02 13:51:39.243609 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:39.243204    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f87c8ddb-fz9jk" event={"ID":"6178871a-a66c-4a02-9382-94eae0fc8ccb","Type":"ContainerDied","Data":"e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7"}
Jul 02 13:51:39.243609 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:39.243268    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f87c8ddb-fz9jk" event={"ID":"6178871a-a66c-4a02-9382-94eae0fc8ccb","Type":"ContainerDied","Data":"14a24f38d4f1fbfdbb23eb3d3b34d10a40b2f79ac0b35379d9007aca0e6dfe55"}
Jul 02 13:51:39.243609 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:39.243288    2574 scope.go:117] "RemoveContainer" containerID="e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7"
Jul 02 13:51:39.252393 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:39.252372    2574 scope.go:117] "RemoveContainer" containerID="e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7"
Jul 02 13:51:39.252656 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:51:39.252636    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7\": container with ID starting with e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7 not found: ID does not exist" containerID="e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7"
Jul 02 13:51:39.252699 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:39.252664    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7"} err="failed to get container status \"e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7\": rpc error: code = NotFound desc = could not find container \"e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7\": container with ID starting with e048150851eb2c68fa641f045a08f03571385326b4b940c680ba78b7ceaaa2f7 not found: ID does not exist"
Jul 02 13:51:39.265677 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:39.265659    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f87c8ddb-fz9jk"]
Jul 02 13:51:39.270297 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:39.270277    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68f87c8ddb-fz9jk"]
Jul 02 13:51:39.351105 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:39.351075    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6178871a-a66c-4a02-9382-94eae0fc8ccb" path="/var/lib/kubelet/pods/6178871a-a66c-4a02-9382-94eae0fc8ccb/volumes"
Jul 02 13:51:52.170112 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:52.170038    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7c747fbc6-wfl57"
Jul 02 13:51:53.191935 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.191904    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-m9fs4"]
Jul 02 13:51:53.192302 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.192206    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6178871a-a66c-4a02-9382-94eae0fc8ccb" containerName="console"
Jul 02 13:51:53.192302 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.192217    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6178871a-a66c-4a02-9382-94eae0fc8ccb" containerName="console"
Jul 02 13:51:53.192302 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.192228    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31110817-ab43-4945-94ce-7cc36a54d1c8" containerName="manager"
Jul 02 13:51:53.192302 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.192233    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="31110817-ab43-4945-94ce-7cc36a54d1c8" containerName="manager"
Jul 02 13:51:53.192429 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.192308    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="31110817-ab43-4945-94ce-7cc36a54d1c8" containerName="manager"
Jul 02 13:51:53.192429 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.192323    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6178871a-a66c-4a02-9382-94eae0fc8ccb" containerName="console"
Jul 02 13:51:53.194273 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.194259    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-m9fs4"
Jul 02 13:51:53.196723 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.196699    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 02 13:51:53.196913 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.196897    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-bhvwv\""
Jul 02 13:51:53.206418 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.206381    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-m9fs4"]
Jul 02 13:51:53.243271 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.243246    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs9b7\" (UniqueName: \"kubernetes.io/projected/534e8181-e835-449f-ad55-0ff46de23e4f-kube-api-access-gs9b7\") pod \"model-serving-api-fd65d7d6-m9fs4\" (UID: \"534e8181-e835-449f-ad55-0ff46de23e4f\") " pod="kserve/model-serving-api-fd65d7d6-m9fs4"
Jul 02 13:51:53.243389 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.243320    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534e8181-e835-449f-ad55-0ff46de23e4f-tls-certs\") pod \"model-serving-api-fd65d7d6-m9fs4\" (UID: \"534e8181-e835-449f-ad55-0ff46de23e4f\") " pod="kserve/model-serving-api-fd65d7d6-m9fs4"
Jul 02 13:51:53.344564 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.344534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534e8181-e835-449f-ad55-0ff46de23e4f-tls-certs\") pod \"model-serving-api-fd65d7d6-m9fs4\" (UID: \"534e8181-e835-449f-ad55-0ff46de23e4f\") " pod="kserve/model-serving-api-fd65d7d6-m9fs4"
Jul 02 13:51:53.344701 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.344599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs9b7\" (UniqueName: \"kubernetes.io/projected/534e8181-e835-449f-ad55-0ff46de23e4f-kube-api-access-gs9b7\") pod \"model-serving-api-fd65d7d6-m9fs4\" (UID: \"534e8181-e835-449f-ad55-0ff46de23e4f\") " pod="kserve/model-serving-api-fd65d7d6-m9fs4"
Jul 02 13:51:53.346935 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.346912    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534e8181-e835-449f-ad55-0ff46de23e4f-tls-certs\") pod \"model-serving-api-fd65d7d6-m9fs4\" (UID: \"534e8181-e835-449f-ad55-0ff46de23e4f\") " pod="kserve/model-serving-api-fd65d7d6-m9fs4"
Jul 02 13:51:53.355456 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.355432    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs9b7\" (UniqueName: \"kubernetes.io/projected/534e8181-e835-449f-ad55-0ff46de23e4f-kube-api-access-gs9b7\") pod \"model-serving-api-fd65d7d6-m9fs4\" (UID: \"534e8181-e835-449f-ad55-0ff46de23e4f\") " pod="kserve/model-serving-api-fd65d7d6-m9fs4"
Jul 02 13:51:53.504018 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.503930    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-m9fs4"
Jul 02 13:51:53.627443 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:53.627421    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-m9fs4"]
Jul 02 13:51:53.629365 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:51:53.629339    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534e8181_e835_449f_ad55_0ff46de23e4f.slice/crio-19309a0de4139d0c2680fdead21ced048ac2f569269b40476d8069d8c5db8776 WatchSource:0}: Error finding container 19309a0de4139d0c2680fdead21ced048ac2f569269b40476d8069d8c5db8776: Status 404 returned error can't find the container with id 19309a0de4139d0c2680fdead21ced048ac2f569269b40476d8069d8c5db8776
Jul 02 13:51:54.305094 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:54.305052    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-m9fs4" event={"ID":"534e8181-e835-449f-ad55-0ff46de23e4f","Type":"ContainerStarted","Data":"19309a0de4139d0c2680fdead21ced048ac2f569269b40476d8069d8c5db8776"}
Jul 02 13:51:56.313769 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:56.313734    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-m9fs4" event={"ID":"534e8181-e835-449f-ad55-0ff46de23e4f","Type":"ContainerStarted","Data":"e379ced12fdf1119812a4b90d99b9457b1323e161a1b7738f892bcabd2bcec87"}
Jul 02 13:51:56.314172 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:56.313903    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-m9fs4"
Jul 02 13:51:56.334230 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:51:56.334187    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-m9fs4" podStartSLOduration=1.2125255990000001 podStartE2EDuration="3.33417453s" totalImagesPullingTime="2.121648931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:51:56.332452118 +0000 UTC m=+453.528075623" watchObservedRunningTime="2026-07-02 13:51:56.33417453 +0000 UTC m=+453.529798048"
Jul 02 13:52:07.322259 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:52:07.322231    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-m9fs4"
Jul 02 13:53:41.456303 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.456273    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"]
Jul 02 13:53:41.458786 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.458767    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:41.461413 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.461394    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kxrcm\""
Jul 02 13:53:41.461529 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.461397    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:53:41.461529 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.461439    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-40523-predictor-serving-cert\""
Jul 02 13:53:41.462669 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.462647    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-40523-kube-rbac-proxy-sar-config\""
Jul 02 13:53:41.462792 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.462705    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:53:41.471121 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.471102    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"]
Jul 02 13:53:41.552332 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.552303    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e1331f3-f255-450a-be3f-18d5eb2a2de9-proxy-tls\") pod \"success-200-isvc-40523-predictor-54dbf486cb-2sn47\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") " pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:41.552469 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.552389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znvtg\" (UniqueName: \"kubernetes.io/projected/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-api-access-znvtg\") pod \"success-200-isvc-40523-predictor-54dbf486cb-2sn47\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") " pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:41.552469 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.552420    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-40523-predictor-54dbf486cb-2sn47\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") " pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:41.653568 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.653536    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znvtg\" (UniqueName: \"kubernetes.io/projected/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-api-access-znvtg\") pod \"success-200-isvc-40523-predictor-54dbf486cb-2sn47\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") " pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:41.653714 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.653577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-40523-predictor-54dbf486cb-2sn47\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") " pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:41.653714 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.653614    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e1331f3-f255-450a-be3f-18d5eb2a2de9-proxy-tls\") pod \"success-200-isvc-40523-predictor-54dbf486cb-2sn47\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") " pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:41.654274 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.654258    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-40523-predictor-54dbf486cb-2sn47\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") " pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:41.656094 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.656068    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e1331f3-f255-450a-be3f-18d5eb2a2de9-proxy-tls\") pod \"success-200-isvc-40523-predictor-54dbf486cb-2sn47\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") " pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:41.662675 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.662649    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znvtg\" (UniqueName: \"kubernetes.io/projected/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-api-access-znvtg\") pod \"success-200-isvc-40523-predictor-54dbf486cb-2sn47\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") " pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:41.768940 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.768843    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:41.905835 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:41.905778    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"]
Jul 02 13:53:41.908093 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:53:41.908064    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1331f3_f255_450a_be3f_18d5eb2a2de9.slice/crio-9dc226f83f167779a0f6e09f458c1b603d05a4d12ed95e87de8e07d165465769 WatchSource:0}: Error finding container 9dc226f83f167779a0f6e09f458c1b603d05a4d12ed95e87de8e07d165465769: Status 404 returned error can't find the container with id 9dc226f83f167779a0f6e09f458c1b603d05a4d12ed95e87de8e07d165465769
Jul 02 13:53:42.747321 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:42.747262    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" event={"ID":"7e1331f3-f255-450a-be3f-18d5eb2a2de9","Type":"ContainerStarted","Data":"9dc226f83f167779a0f6e09f458c1b603d05a4d12ed95e87de8e07d165465769"}
Jul 02 13:53:54.818589 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:54.818542    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" event={"ID":"7e1331f3-f255-450a-be3f-18d5eb2a2de9","Type":"ContainerStarted","Data":"6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53"}
Jul 02 13:53:56.831213 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:56.831108    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" event={"ID":"7e1331f3-f255-450a-be3f-18d5eb2a2de9","Type":"ContainerStarted","Data":"5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6"}
Jul 02 13:53:56.831647 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:56.831311    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:56.831647 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:56.831460    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:53:56.832564 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:56.832537    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:53:56.858067 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:56.858025    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" podStartSLOduration=1.303454358 podStartE2EDuration="15.858011036s" totalImagesPullingTime="14.554556678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:53:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:53:56.853783769 +0000 UTC m=+574.049407278" watchObservedRunningTime="2026-07-02 13:53:56.858011036 +0000 UTC m=+574.053634539"
Jul 02 13:53:57.835757 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:53:57.835716    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:54:02.840124 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:54:02.840098    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:54:02.840603 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:54:02.840511    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:54:12.840651 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:54:12.840611    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:54:22.841224 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:54:22.841190    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:54:23.308149 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:54:23.308106    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 13:54:23.308308 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:54:23.308293    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 13:54:23.314501 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:54:23.314481    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 13:54:23.314623 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:54:23.314591    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 13:54:32.840741 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:54:32.840705    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:54:42.841325 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:54:42.841291    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:55:01.490290 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:01.490215    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"]
Jul 02 13:55:01.492721 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:01.492704    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:01.495484 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:01.495456    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-40523-kube-rbac-proxy-sar-config\""
Jul 02 13:55:01.495647 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:01.495469    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-40523-serving-cert\""
Jul 02 13:55:01.502373 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:01.502345    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"]
Jul 02 13:55:01.514345 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:01.514313    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3013f16b-01cc-436e-be2a-691c507712f6-openshift-service-ca-bundle\") pod \"switch-graph-40523-6b6759674-ksx98\" (UID: \"3013f16b-01cc-436e-be2a-691c507712f6\") " pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:01.514472 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:01.514446    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3013f16b-01cc-436e-be2a-691c507712f6-proxy-tls\") pod \"switch-graph-40523-6b6759674-ksx98\" (UID: \"3013f16b-01cc-436e-be2a-691c507712f6\") " pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:01.614797 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:01.614766    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3013f16b-01cc-436e-be2a-691c507712f6-proxy-tls\") pod \"switch-graph-40523-6b6759674-ksx98\" (UID: \"3013f16b-01cc-436e-be2a-691c507712f6\") " pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:01.614980 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:01.614829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3013f16b-01cc-436e-be2a-691c507712f6-openshift-service-ca-bundle\") pod \"switch-graph-40523-6b6759674-ksx98\" (UID: \"3013f16b-01cc-436e-be2a-691c507712f6\") " pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:01.614980 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:55:01.614939    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-40523-serving-cert: secret "switch-graph-40523-serving-cert" not found
Jul 02 13:55:01.615068 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:55:01.615019    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3013f16b-01cc-436e-be2a-691c507712f6-proxy-tls podName:3013f16b-01cc-436e-be2a-691c507712f6 nodeName:}" failed. No retries permitted until 2026-07-02 13:55:02.11499683 +0000 UTC m=+639.310620315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3013f16b-01cc-436e-be2a-691c507712f6-proxy-tls") pod "switch-graph-40523-6b6759674-ksx98" (UID: "3013f16b-01cc-436e-be2a-691c507712f6") : secret "switch-graph-40523-serving-cert" not found
Jul 02 13:55:01.615700 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:01.615674    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3013f16b-01cc-436e-be2a-691c507712f6-openshift-service-ca-bundle\") pod \"switch-graph-40523-6b6759674-ksx98\" (UID: \"3013f16b-01cc-436e-be2a-691c507712f6\") " pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:02.119079 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:02.119044    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3013f16b-01cc-436e-be2a-691c507712f6-proxy-tls\") pod \"switch-graph-40523-6b6759674-ksx98\" (UID: \"3013f16b-01cc-436e-be2a-691c507712f6\") " pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:02.121521 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:02.121491    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3013f16b-01cc-436e-be2a-691c507712f6-proxy-tls\") pod \"switch-graph-40523-6b6759674-ksx98\" (UID: \"3013f16b-01cc-436e-be2a-691c507712f6\") " pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:02.403704 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:02.403626    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:02.531540 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:02.531420    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"]
Jul 02 13:55:02.533763 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:55:02.533731    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3013f16b_01cc_436e_be2a_691c507712f6.slice/crio-311299e4a067f8a9f90ebd97cbf53c385ef8ec2311f07e7d20f77f25a16b8f64 WatchSource:0}: Error finding container 311299e4a067f8a9f90ebd97cbf53c385ef8ec2311f07e7d20f77f25a16b8f64: Status 404 returned error can't find the container with id 311299e4a067f8a9f90ebd97cbf53c385ef8ec2311f07e7d20f77f25a16b8f64
Jul 02 13:55:02.535583 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:02.535568    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:55:03.108278 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:03.108243    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" event={"ID":"3013f16b-01cc-436e-be2a-691c507712f6","Type":"ContainerStarted","Data":"311299e4a067f8a9f90ebd97cbf53c385ef8ec2311f07e7d20f77f25a16b8f64"}
Jul 02 13:55:06.123284 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:06.123251    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" event={"ID":"3013f16b-01cc-436e-be2a-691c507712f6","Type":"ContainerStarted","Data":"48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721"}
Jul 02 13:55:06.123682 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:06.123311    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:06.142820 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:06.142772    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" podStartSLOduration=2.424083937 podStartE2EDuration="5.142757158s" totalImagesPullingTime="2.718673221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:55:06.14203347 +0000 UTC m=+643.337656980" watchObservedRunningTime="2026-07-02 13:55:06.142757158 +0000 UTC m=+643.338380664"
Jul 02 13:55:12.134558 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:12.134531    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:15.607077 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.607038    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"]
Jul 02 13:55:15.607513 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.607321    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" podUID="3013f16b-01cc-436e-be2a-691c507712f6" containerName="switch-graph-40523" containerID="cri-o://48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721" gracePeriod=30
Jul 02 13:55:15.843800 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.843770    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"]
Jul 02 13:55:15.846667 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.846644    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:15.849070 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.849051    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-68692-predictor-serving-cert\""
Jul 02 13:55:15.849315 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.849296    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-68692-kube-rbac-proxy-sar-config\""
Jul 02 13:55:15.864248 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.864187    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"]
Jul 02 13:55:15.865730 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.865706    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"]
Jul 02 13:55:15.865958 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.865923    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kserve-container" containerID="cri-o://6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53" gracePeriod=30
Jul 02 13:55:15.866068 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.866020    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kube-rbac-proxy" containerID="cri-o://5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6" gracePeriod=30
Jul 02 13:55:15.931247 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.931208    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c86d8\" (UniqueName: \"kubernetes.io/projected/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-api-access-c86d8\") pod \"success-200-isvc-68692-predictor-98ddcc858-6r58m\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") " pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:15.931381 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.931317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-proxy-tls\") pod \"success-200-isvc-68692-predictor-98ddcc858-6r58m\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") " pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:15.931381 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:15.931365    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-68692-predictor-98ddcc858-6r58m\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") " pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:16.032261 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:16.032227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-68692-predictor-98ddcc858-6r58m\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") " pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:16.032422 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:16.032293    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c86d8\" (UniqueName: \"kubernetes.io/projected/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-api-access-c86d8\") pod \"success-200-isvc-68692-predictor-98ddcc858-6r58m\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") " pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:16.032422 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:16.032346    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-proxy-tls\") pod \"success-200-isvc-68692-predictor-98ddcc858-6r58m\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") " pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:16.032597 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:55:16.032493    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-68692-predictor-serving-cert: secret "success-200-isvc-68692-predictor-serving-cert" not found
Jul 02 13:55:16.032597 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:55:16.032573    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-proxy-tls podName:0dd64166-d6c9-4373-ac9b-b5d692fd6f62 nodeName:}" failed. No retries permitted until 2026-07-02 13:55:16.5325503 +0000 UTC m=+653.728173789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-proxy-tls") pod "success-200-isvc-68692-predictor-98ddcc858-6r58m" (UID: "0dd64166-d6c9-4373-ac9b-b5d692fd6f62") : secret "success-200-isvc-68692-predictor-serving-cert" not found
Jul 02 13:55:16.032956 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:16.032938    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-68692-predictor-98ddcc858-6r58m\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") " pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:16.041564 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:16.041535    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c86d8\" (UniqueName: \"kubernetes.io/projected/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-api-access-c86d8\") pod \"success-200-isvc-68692-predictor-98ddcc858-6r58m\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") " pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:16.166444 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:16.166413    2574 generic.go:356] "Generic (PLEG): container finished" podID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerID="5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6" exitCode=2
Jul 02 13:55:16.166603 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:16.166494    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" event={"ID":"7e1331f3-f255-450a-be3f-18d5eb2a2de9","Type":"ContainerDied","Data":"5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6"}
Jul 02 13:55:16.537739 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:16.537660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-proxy-tls\") pod \"success-200-isvc-68692-predictor-98ddcc858-6r58m\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") " pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:16.539975 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:16.539944    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-proxy-tls\") pod \"success-200-isvc-68692-predictor-98ddcc858-6r58m\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") " pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:16.757994 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:16.757966    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:16.887723 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:16.887697    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"]
Jul 02 13:55:16.890343 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:55:16.890312    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd64166_d6c9_4373_ac9b_b5d692fd6f62.slice/crio-c86b350b1746c40e3ac2cb4ff6d8deb67bd9a5d7e11670b235ec9135a73cc4d0 WatchSource:0}: Error finding container c86b350b1746c40e3ac2cb4ff6d8deb67bd9a5d7e11670b235ec9135a73cc4d0: Status 404 returned error can't find the container with id c86b350b1746c40e3ac2cb4ff6d8deb67bd9a5d7e11670b235ec9135a73cc4d0
Jul 02 13:55:17.132903 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:17.132801    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" podUID="3013f16b-01cc-436e-be2a-691c507712f6" containerName="switch-graph-40523" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:17.172277 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:17.172248    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" event={"ID":"0dd64166-d6c9-4373-ac9b-b5d692fd6f62","Type":"ContainerStarted","Data":"2a86d0622af4e7c6154f683c8fd20d18f73207503234fca4754c8dea198fc369"}
Jul 02 13:55:17.172453 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:17.172309    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:17.172453 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:17.172330    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" event={"ID":"0dd64166-d6c9-4373-ac9b-b5d692fd6f62","Type":"ContainerStarted","Data":"2f153675dc18a2a23a7da747920b6803f858a4b13925d85b1e80045c31773365"}
Jul 02 13:55:17.172453 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:17.172350    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" event={"ID":"0dd64166-d6c9-4373-ac9b-b5d692fd6f62","Type":"ContainerStarted","Data":"c86b350b1746c40e3ac2cb4ff6d8deb67bd9a5d7e11670b235ec9135a73cc4d0"}
Jul 02 13:55:17.172453 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:17.172371    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:17.173898 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:17.173874    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:55:17.197781 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:17.197730    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" podStartSLOduration=2.197714533 podStartE2EDuration="2.197714533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:55:17.196362333 +0000 UTC m=+654.391985839" watchObservedRunningTime="2026-07-02 13:55:17.197714533 +0000 UTC m=+654.393338039"
Jul 02 13:55:17.836489 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:17.836452    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 02 13:55:18.177077 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:18.177045    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:55:22.015051 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.015028    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:55:22.085202 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.085120    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e1331f3-f255-450a-be3f-18d5eb2a2de9-proxy-tls\") pod \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") "
Jul 02 13:55:22.085202 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.085170    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znvtg\" (UniqueName: \"kubernetes.io/projected/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-api-access-znvtg\") pod \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") "
Jul 02 13:55:22.085357 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.085239    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-rbac-proxy-sar-config\") pod \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\" (UID: \"7e1331f3-f255-450a-be3f-18d5eb2a2de9\") "
Jul 02 13:55:22.085632 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.085607    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e1331f3-f255-450a-be3f-18d5eb2a2de9" (UID: "7e1331f3-f255-450a-be3f-18d5eb2a2de9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:55:22.087217 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.087189    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1331f3-f255-450a-be3f-18d5eb2a2de9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e1331f3-f255-450a-be3f-18d5eb2a2de9" (UID: "7e1331f3-f255-450a-be3f-18d5eb2a2de9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:55:22.087318 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.087251    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-api-access-znvtg" (OuterVolumeSpecName: "kube-api-access-znvtg") pod "7e1331f3-f255-450a-be3f-18d5eb2a2de9" (UID: "7e1331f3-f255-450a-be3f-18d5eb2a2de9"). InnerVolumeSpecName "kube-api-access-znvtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:55:22.133041 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.133001    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" podUID="3013f16b-01cc-436e-be2a-691c507712f6" containerName="switch-graph-40523" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:22.186174 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.186151    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e1331f3-f255-450a-be3f-18d5eb2a2de9-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:55:22.186174 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.186172    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znvtg\" (UniqueName: \"kubernetes.io/projected/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-api-access-znvtg\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:55:22.186306 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.186182    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e1331f3-f255-450a-be3f-18d5eb2a2de9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:55:22.198840 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.198815    2574 generic.go:356] "Generic (PLEG): container finished" podID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerID="6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53" exitCode=0
Jul 02 13:55:22.198940 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.198864    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"
Jul 02 13:55:22.198940 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.198873    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" event={"ID":"7e1331f3-f255-450a-be3f-18d5eb2a2de9","Type":"ContainerDied","Data":"6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53"}
Jul 02 13:55:22.198940 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.198919    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47" event={"ID":"7e1331f3-f255-450a-be3f-18d5eb2a2de9","Type":"ContainerDied","Data":"9dc226f83f167779a0f6e09f458c1b603d05a4d12ed95e87de8e07d165465769"}
Jul 02 13:55:22.199045 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.198945    2574 scope.go:117] "RemoveContainer" containerID="5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6"
Jul 02 13:55:22.206942 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.206928    2574 scope.go:117] "RemoveContainer" containerID="6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53"
Jul 02 13:55:22.214018 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.213998    2574 scope.go:117] "RemoveContainer" containerID="5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6"
Jul 02 13:55:22.214266 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:55:22.214250    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6\": container with ID starting with 5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6 not found: ID does not exist" containerID="5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6"
Jul 02 13:55:22.214317 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.214276    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6"} err="failed to get container status \"5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6\": rpc error: code = NotFound desc = could not find container \"5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6\": container with ID starting with 5ba21798885f487b622e961ca08af8646f9e256b8795b92ce23e6ec37036fcf6 not found: ID does not exist"
Jul 02 13:55:22.214317 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.214292    2574 scope.go:117] "RemoveContainer" containerID="6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53"
Jul 02 13:55:22.214500 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:55:22.214485    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53\": container with ID starting with 6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53 not found: ID does not exist" containerID="6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53"
Jul 02 13:55:22.214535 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.214506    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53"} err="failed to get container status \"6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53\": rpc error: code = NotFound desc = could not find container \"6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53\": container with ID starting with 6a8a27139d233f6b01597e7f9efc0c6ba0e20f6e28cfaa3e2a880c608a3d6c53 not found: ID does not exist"
Jul 02 13:55:22.221499 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.221475    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"]
Jul 02 13:55:22.226183 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:22.226160    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40523-predictor-54dbf486cb-2sn47"]
Jul 02 13:55:23.182030 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:23.182000    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:55:23.182521 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:23.182494    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:55:23.350425 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:23.350396    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" path="/var/lib/kubelet/pods/7e1331f3-f255-450a-be3f-18d5eb2a2de9/volumes"
Jul 02 13:55:27.131982 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:27.131939    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" podUID="3013f16b-01cc-436e-be2a-691c507712f6" containerName="switch-graph-40523" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:27.132372 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:27.132059    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:32.132505 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:32.132459    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" podUID="3013f16b-01cc-436e-be2a-691c507712f6" containerName="switch-graph-40523" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:33.182824 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:33.182785    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:55:37.131996 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:37.131956    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" podUID="3013f16b-01cc-436e-be2a-691c507712f6" containerName="switch-graph-40523" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:42.131703 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:42.131663    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" podUID="3013f16b-01cc-436e-be2a-691c507712f6" containerName="switch-graph-40523" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:55:43.183091 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:43.183050    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:55:45.797938 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:45.797913    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:45.891153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:45.891045    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3013f16b-01cc-436e-be2a-691c507712f6-openshift-service-ca-bundle\") pod \"3013f16b-01cc-436e-be2a-691c507712f6\" (UID: \"3013f16b-01cc-436e-be2a-691c507712f6\") "
Jul 02 13:55:45.891153 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:45.891115    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3013f16b-01cc-436e-be2a-691c507712f6-proxy-tls\") pod \"3013f16b-01cc-436e-be2a-691c507712f6\" (UID: \"3013f16b-01cc-436e-be2a-691c507712f6\") "
Jul 02 13:55:45.891426 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:45.891402    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3013f16b-01cc-436e-be2a-691c507712f6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3013f16b-01cc-436e-be2a-691c507712f6" (UID: "3013f16b-01cc-436e-be2a-691c507712f6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:55:45.893082 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:45.893062    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3013f16b-01cc-436e-be2a-691c507712f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3013f16b-01cc-436e-be2a-691c507712f6" (UID: "3013f16b-01cc-436e-be2a-691c507712f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:55:45.992106 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:45.992079    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3013f16b-01cc-436e-be2a-691c507712f6-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:55:45.992106 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:45.992103    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3013f16b-01cc-436e-be2a-691c507712f6-openshift-service-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:55:46.309994 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:46.309965    2574 generic.go:356] "Generic (PLEG): container finished" podID="3013f16b-01cc-436e-be2a-691c507712f6" containerID="48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721" exitCode=137
Jul 02 13:55:46.310166 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:46.310008    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"
Jul 02 13:55:46.310166 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:46.310010    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" event={"ID":"3013f16b-01cc-436e-be2a-691c507712f6","Type":"ContainerDied","Data":"48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721"}
Jul 02 13:55:46.310259 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:46.310157    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98" event={"ID":"3013f16b-01cc-436e-be2a-691c507712f6","Type":"ContainerDied","Data":"311299e4a067f8a9f90ebd97cbf53c385ef8ec2311f07e7d20f77f25a16b8f64"}
Jul 02 13:55:46.310259 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:46.310193    2574 scope.go:117] "RemoveContainer" containerID="48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721"
Jul 02 13:55:46.319320 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:46.319304    2574 scope.go:117] "RemoveContainer" containerID="48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721"
Jul 02 13:55:46.319578 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:55:46.319560    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721\": container with ID starting with 48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721 not found: ID does not exist" containerID="48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721"
Jul 02 13:55:46.319637 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:46.319588    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721"} err="failed to get container status \"48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721\": rpc error: code = NotFound desc = could not find container \"48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721\": container with ID starting with 48f6a5436c49f70ee058b4eade9e95f83899bc0be5f9dfad4eb541e6c3b7b721 not found: ID does not exist"
Jul 02 13:55:46.335773 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:46.335743    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"]
Jul 02 13:55:46.339884 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:46.339862    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-40523-6b6759674-ksx98"]
Jul 02 13:55:47.350354 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:47.350320    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3013f16b-01cc-436e-be2a-691c507712f6" path="/var/lib/kubelet/pods/3013f16b-01cc-436e-be2a-691c507712f6/volumes"
Jul 02 13:55:53.183011 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:55:53.182972    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:56:01.551773 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.551736    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"]
Jul 02 13:56:01.552230 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.552124    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kube-rbac-proxy"
Jul 02 13:56:01.552230 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.552156    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kube-rbac-proxy"
Jul 02 13:56:01.552230 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.552179    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kserve-container"
Jul 02 13:56:01.552230 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.552185    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kserve-container"
Jul 02 13:56:01.552230 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.552193    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3013f16b-01cc-436e-be2a-691c507712f6" containerName="switch-graph-40523"
Jul 02 13:56:01.552230 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.552199    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3013f16b-01cc-436e-be2a-691c507712f6" containerName="switch-graph-40523"
Jul 02 13:56:01.552491 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.552277    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3013f16b-01cc-436e-be2a-691c507712f6" containerName="switch-graph-40523"
Jul 02 13:56:01.552491 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.552291    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kube-rbac-proxy"
Jul 02 13:56:01.552491 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.552302    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1331f3-f255-450a-be3f-18d5eb2a2de9" containerName="kserve-container"
Jul 02 13:56:01.554688 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.554672    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:01.557803 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.557780    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 02 13:56:01.557916 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.557826    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 02 13:56:01.565102 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.565081    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"]
Jul 02 13:56:01.601147 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.601097    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/824b6be7-7681-43c0-9268-95b4cfca312e-proxy-tls\") pod \"model-chainer-59c85f56c9-2z4kl\" (UID: \"824b6be7-7681-43c0-9268-95b4cfca312e\") " pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:01.601315 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.601154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/824b6be7-7681-43c0-9268-95b4cfca312e-openshift-service-ca-bundle\") pod \"model-chainer-59c85f56c9-2z4kl\" (UID: \"824b6be7-7681-43c0-9268-95b4cfca312e\") " pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:01.701938 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.701895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/824b6be7-7681-43c0-9268-95b4cfca312e-proxy-tls\") pod \"model-chainer-59c85f56c9-2z4kl\" (UID: \"824b6be7-7681-43c0-9268-95b4cfca312e\") " pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:01.701938 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.701940    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/824b6be7-7681-43c0-9268-95b4cfca312e-openshift-service-ca-bundle\") pod \"model-chainer-59c85f56c9-2z4kl\" (UID: \"824b6be7-7681-43c0-9268-95b4cfca312e\") " pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:01.702171 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:56:01.702045    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 02 13:56:01.702171 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:56:01.702110    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/824b6be7-7681-43c0-9268-95b4cfca312e-proxy-tls podName:824b6be7-7681-43c0-9268-95b4cfca312e nodeName:}" failed. No retries permitted until 2026-07-02 13:56:02.202091681 +0000 UTC m=+699.397715171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/824b6be7-7681-43c0-9268-95b4cfca312e-proxy-tls") pod "model-chainer-59c85f56c9-2z4kl" (UID: "824b6be7-7681-43c0-9268-95b4cfca312e") : secret "model-chainer-serving-cert" not found
Jul 02 13:56:01.702575 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:01.702555    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/824b6be7-7681-43c0-9268-95b4cfca312e-openshift-service-ca-bundle\") pod \"model-chainer-59c85f56c9-2z4kl\" (UID: \"824b6be7-7681-43c0-9268-95b4cfca312e\") " pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:02.207399 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:02.207363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/824b6be7-7681-43c0-9268-95b4cfca312e-proxy-tls\") pod \"model-chainer-59c85f56c9-2z4kl\" (UID: \"824b6be7-7681-43c0-9268-95b4cfca312e\") " pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:02.209668 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:02.209647    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/824b6be7-7681-43c0-9268-95b4cfca312e-proxy-tls\") pod \"model-chainer-59c85f56c9-2z4kl\" (UID: \"824b6be7-7681-43c0-9268-95b4cfca312e\") " pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:02.465758 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:02.465679    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:02.593852 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:02.593829    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"]
Jul 02 13:56:02.596015 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:56:02.595978    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824b6be7_7681_43c0_9268_95b4cfca312e.slice/crio-54cfb25f2e69765f64487ddfc7d7a46b1fe2005fb9b4bca47fff0225c25a9c3f WatchSource:0}: Error finding container 54cfb25f2e69765f64487ddfc7d7a46b1fe2005fb9b4bca47fff0225c25a9c3f: Status 404 returned error can't find the container with id 54cfb25f2e69765f64487ddfc7d7a46b1fe2005fb9b4bca47fff0225c25a9c3f
Jul 02 13:56:03.184042 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:03.184008    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 13:56:03.387856 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:03.387825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" event={"ID":"824b6be7-7681-43c0-9268-95b4cfca312e","Type":"ContainerStarted","Data":"01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f"}
Jul 02 13:56:03.388012 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:03.387871    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" event={"ID":"824b6be7-7681-43c0-9268-95b4cfca312e","Type":"ContainerStarted","Data":"54cfb25f2e69765f64487ddfc7d7a46b1fe2005fb9b4bca47fff0225c25a9c3f"}
Jul 02 13:56:03.388012 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:03.387971    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:03.407095 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:03.407053    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" podStartSLOduration=2.40704079 podStartE2EDuration="2.40704079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:56:03.406273368 +0000 UTC m=+700.601896878" watchObservedRunningTime="2026-07-02 13:56:03.40704079 +0000 UTC m=+700.602664298"
Jul 02 13:56:09.397738 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:09.397711    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:11.702199 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:11.702162    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"]
Jul 02 13:56:11.702585 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:11.702350    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" podUID="824b6be7-7681-43c0-9268-95b4cfca312e" containerName="model-chainer" containerID="cri-o://01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f" gracePeriod=30
Jul 02 13:56:11.879923 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:11.879884    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"]
Jul 02 13:56:11.882687 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:11.882668    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:11.885214 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:11.885189    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-88827-predictor-serving-cert\""
Jul 02 13:56:11.885338 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:11.885222    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-88827-kube-rbac-proxy-sar-config\""
Jul 02 13:56:11.895728 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:11.895709    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"]
Jul 02 13:56:11.991713 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:11.991625    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dt2x\" (UniqueName: \"kubernetes.io/projected/71af5689-176e-4d90-8d89-d5a70a978f78-kube-api-access-6dt2x\") pod \"success-200-isvc-88827-predictor-68bb654446-xn6vz\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") " pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:11.991713 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:11.991657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71af5689-176e-4d90-8d89-d5a70a978f78-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-88827-predictor-68bb654446-xn6vz\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") " pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:11.991713 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:11.991692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71af5689-176e-4d90-8d89-d5a70a978f78-proxy-tls\") pod \"success-200-isvc-88827-predictor-68bb654446-xn6vz\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") " pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:12.092396 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:12.092363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71af5689-176e-4d90-8d89-d5a70a978f78-proxy-tls\") pod \"success-200-isvc-88827-predictor-68bb654446-xn6vz\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") " pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:12.092564 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:12.092429    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dt2x\" (UniqueName: \"kubernetes.io/projected/71af5689-176e-4d90-8d89-d5a70a978f78-kube-api-access-6dt2x\") pod \"success-200-isvc-88827-predictor-68bb654446-xn6vz\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") " pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:12.092564 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:12.092448    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71af5689-176e-4d90-8d89-d5a70a978f78-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-88827-predictor-68bb654446-xn6vz\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") " pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:12.092564 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:56:12.092522    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-88827-predictor-serving-cert: secret "success-200-isvc-88827-predictor-serving-cert" not found
Jul 02 13:56:12.092724 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:56:12.092603    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71af5689-176e-4d90-8d89-d5a70a978f78-proxy-tls podName:71af5689-176e-4d90-8d89-d5a70a978f78 nodeName:}" failed. No retries permitted until 2026-07-02 13:56:12.592578716 +0000 UTC m=+709.788202206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71af5689-176e-4d90-8d89-d5a70a978f78-proxy-tls") pod "success-200-isvc-88827-predictor-68bb654446-xn6vz" (UID: "71af5689-176e-4d90-8d89-d5a70a978f78") : secret "success-200-isvc-88827-predictor-serving-cert" not found
Jul 02 13:56:12.093052 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:12.093034    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71af5689-176e-4d90-8d89-d5a70a978f78-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-88827-predictor-68bb654446-xn6vz\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") " pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:12.105551 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:12.105525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dt2x\" (UniqueName: \"kubernetes.io/projected/71af5689-176e-4d90-8d89-d5a70a978f78-kube-api-access-6dt2x\") pod \"success-200-isvc-88827-predictor-68bb654446-xn6vz\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") " pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:12.596441 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:12.596398    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71af5689-176e-4d90-8d89-d5a70a978f78-proxy-tls\") pod \"success-200-isvc-88827-predictor-68bb654446-xn6vz\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") " pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:12.598768 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:12.598734    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71af5689-176e-4d90-8d89-d5a70a978f78-proxy-tls\") pod \"success-200-isvc-88827-predictor-68bb654446-xn6vz\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") " pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:12.794625 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:12.794594    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:12.926482 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:12.926462    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"]
Jul 02 13:56:12.928730 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:56:12.928710    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71af5689_176e_4d90_8d89_d5a70a978f78.slice/crio-0aacad459935d4071ea9ef1e17352b971f0b404c53a921648b0ff5d9820e6922 WatchSource:0}: Error finding container 0aacad459935d4071ea9ef1e17352b971f0b404c53a921648b0ff5d9820e6922: Status 404 returned error can't find the container with id 0aacad459935d4071ea9ef1e17352b971f0b404c53a921648b0ff5d9820e6922
Jul 02 13:56:13.434004 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:13.433961    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" event={"ID":"71af5689-176e-4d90-8d89-d5a70a978f78","Type":"ContainerStarted","Data":"0875218296ed58854097ccd311e7ab88c57a76eba33112c40daf1fdecf563018"}
Jul 02 13:56:13.434215 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:13.434026    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" event={"ID":"71af5689-176e-4d90-8d89-d5a70a978f78","Type":"ContainerStarted","Data":"eae4ed013b8b5787382bdd4e15b8e12a2d3383258d5ae9a07cf9148f1064d6a4"}
Jul 02 13:56:13.434215 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:13.434059    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:13.434215 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:13.434087    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" event={"ID":"71af5689-176e-4d90-8d89-d5a70a978f78","Type":"ContainerStarted","Data":"0aacad459935d4071ea9ef1e17352b971f0b404c53a921648b0ff5d9820e6922"}
Jul 02 13:56:13.434393 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:13.434234    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:13.435422 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:13.435397    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:56:13.458695 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:13.458656    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" podStartSLOduration=2.458643743 podStartE2EDuration="2.458643743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:56:13.458084281 +0000 UTC m=+710.653707800" watchObservedRunningTime="2026-07-02 13:56:13.458643743 +0000 UTC m=+710.654267246"
Jul 02 13:56:14.397114 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:14.397020    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" podUID="824b6be7-7681-43c0-9268-95b4cfca312e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:56:14.438519 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:14.438475    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:56:15.885286 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:15.885256    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"]
Jul 02 13:56:15.887642 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:15.887626    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 13:56:15.893046 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:15.893021    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-68692-serving-cert\""
Jul 02 13:56:15.893193 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:15.893024    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-68692-kube-rbac-proxy-sar-config\""
Jul 02 13:56:15.899113 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:15.899094    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"]
Jul 02 13:56:15.926408 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:15.926383    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e56553-8dc5-47e1-9355-cbf353a81268-proxy-tls\") pod \"switch-graph-68692-78776d6ccd-fsth8\" (UID: \"b7e56553-8dc5-47e1-9355-cbf353a81268\") " pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 13:56:15.926523 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:15.926435    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e56553-8dc5-47e1-9355-cbf353a81268-openshift-service-ca-bundle\") pod \"switch-graph-68692-78776d6ccd-fsth8\" (UID: \"b7e56553-8dc5-47e1-9355-cbf353a81268\") " pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 13:56:16.027783 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:16.027755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e56553-8dc5-47e1-9355-cbf353a81268-proxy-tls\") pod \"switch-graph-68692-78776d6ccd-fsth8\" (UID: \"b7e56553-8dc5-47e1-9355-cbf353a81268\") " pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 13:56:16.027911 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:16.027815    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e56553-8dc5-47e1-9355-cbf353a81268-openshift-service-ca-bundle\") pod \"switch-graph-68692-78776d6ccd-fsth8\" (UID: \"b7e56553-8dc5-47e1-9355-cbf353a81268\") " pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 13:56:16.027911 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:56:16.027902    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-68692-serving-cert: secret "switch-graph-68692-serving-cert" not found
Jul 02 13:56:16.027979 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:56:16.027959    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7e56553-8dc5-47e1-9355-cbf353a81268-proxy-tls podName:b7e56553-8dc5-47e1-9355-cbf353a81268 nodeName:}" failed. No retries permitted until 2026-07-02 13:56:16.527944015 +0000 UTC m=+713.723567504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b7e56553-8dc5-47e1-9355-cbf353a81268-proxy-tls") pod "switch-graph-68692-78776d6ccd-fsth8" (UID: "b7e56553-8dc5-47e1-9355-cbf353a81268") : secret "switch-graph-68692-serving-cert" not found
Jul 02 13:56:16.028453 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:16.028436    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e56553-8dc5-47e1-9355-cbf353a81268-openshift-service-ca-bundle\") pod \"switch-graph-68692-78776d6ccd-fsth8\" (UID: \"b7e56553-8dc5-47e1-9355-cbf353a81268\") " pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 13:56:16.532214 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:16.532178    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e56553-8dc5-47e1-9355-cbf353a81268-proxy-tls\") pod \"switch-graph-68692-78776d6ccd-fsth8\" (UID: \"b7e56553-8dc5-47e1-9355-cbf353a81268\") " pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 13:56:16.534531 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:16.534512    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e56553-8dc5-47e1-9355-cbf353a81268-proxy-tls\") pod \"switch-graph-68692-78776d6ccd-fsth8\" (UID: \"b7e56553-8dc5-47e1-9355-cbf353a81268\") " pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 13:56:16.797819 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:16.797736    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 13:56:16.931339 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:16.931278    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"]
Jul 02 13:56:16.934954 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:56:16.934927    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e56553_8dc5_47e1_9355_cbf353a81268.slice/crio-76d8422ed9c26887f494b7a90d409823fcc7e1f60fa1d3eb7d9ba0686136ba5f WatchSource:0}: Error finding container 76d8422ed9c26887f494b7a90d409823fcc7e1f60fa1d3eb7d9ba0686136ba5f: Status 404 returned error can't find the container with id 76d8422ed9c26887f494b7a90d409823fcc7e1f60fa1d3eb7d9ba0686136ba5f
Jul 02 13:56:17.453052 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:17.453020    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" event={"ID":"b7e56553-8dc5-47e1-9355-cbf353a81268","Type":"ContainerStarted","Data":"f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756"}
Jul 02 13:56:17.453223 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:17.453070    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" event={"ID":"b7e56553-8dc5-47e1-9355-cbf353a81268","Type":"ContainerStarted","Data":"76d8422ed9c26887f494b7a90d409823fcc7e1f60fa1d3eb7d9ba0686136ba5f"}
Jul 02 13:56:17.453223 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:17.453126    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 13:56:17.471816 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:17.471767    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" podStartSLOduration=2.471752483 podStartE2EDuration="2.471752483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:56:17.470741784 +0000 UTC m=+714.666365302" watchObservedRunningTime="2026-07-02 13:56:17.471752483 +0000 UTC m=+714.667375989"
Jul 02 13:56:19.396000 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:19.395966    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" podUID="824b6be7-7681-43c0-9268-95b4cfca312e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:56:19.443322 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:19.443295    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:56:19.443769 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:19.443746    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:56:23.462868 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:23.462837    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 13:56:24.396360 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:24.396324    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" podUID="824b6be7-7681-43c0-9268-95b4cfca312e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:56:24.396601 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:24.396426    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:29.396156 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:29.396101    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" podUID="824b6be7-7681-43c0-9268-95b4cfca312e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:56:29.443946 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:29.443907    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:56:34.396659 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:34.396624    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" podUID="824b6be7-7681-43c0-9268-95b4cfca312e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:56:39.396593 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:39.396553    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" podUID="824b6be7-7681-43c0-9268-95b4cfca312e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:56:39.444283 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:39.444248    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:56:42.356942 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.356912    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:42.456615 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.456586    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/824b6be7-7681-43c0-9268-95b4cfca312e-proxy-tls\") pod \"824b6be7-7681-43c0-9268-95b4cfca312e\" (UID: \"824b6be7-7681-43c0-9268-95b4cfca312e\") "
Jul 02 13:56:42.456762 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.456667    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/824b6be7-7681-43c0-9268-95b4cfca312e-openshift-service-ca-bundle\") pod \"824b6be7-7681-43c0-9268-95b4cfca312e\" (UID: \"824b6be7-7681-43c0-9268-95b4cfca312e\") "
Jul 02 13:56:42.457009 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.456981    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824b6be7-7681-43c0-9268-95b4cfca312e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "824b6be7-7681-43c0-9268-95b4cfca312e" (UID: "824b6be7-7681-43c0-9268-95b4cfca312e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:56:42.458433 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.458413    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824b6be7-7681-43c0-9268-95b4cfca312e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "824b6be7-7681-43c0-9268-95b4cfca312e" (UID: "824b6be7-7681-43c0-9268-95b4cfca312e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:56:42.557710 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.557651    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/824b6be7-7681-43c0-9268-95b4cfca312e-openshift-service-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:56:42.557710 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.557671    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/824b6be7-7681-43c0-9268-95b4cfca312e-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 13:56:42.561295 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.561272    2574 generic.go:356] "Generic (PLEG): container finished" podID="824b6be7-7681-43c0-9268-95b4cfca312e" containerID="01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f" exitCode=0
Jul 02 13:56:42.561393 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.561316    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"
Jul 02 13:56:42.561393 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.561324    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" event={"ID":"824b6be7-7681-43c0-9268-95b4cfca312e","Type":"ContainerDied","Data":"01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f"}
Jul 02 13:56:42.561462 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.561393    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl" event={"ID":"824b6be7-7681-43c0-9268-95b4cfca312e","Type":"ContainerDied","Data":"54cfb25f2e69765f64487ddfc7d7a46b1fe2005fb9b4bca47fff0225c25a9c3f"}
Jul 02 13:56:42.561462 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.561417    2574 scope.go:117] "RemoveContainer" containerID="01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f"
Jul 02 13:56:42.569867 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.569851    2574 scope.go:117] "RemoveContainer" containerID="01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f"
Jul 02 13:56:42.570106 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:56:42.570089    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f\": container with ID starting with 01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f not found: ID does not exist" containerID="01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f"
Jul 02 13:56:42.570172 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.570116    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f"} err="failed to get container status \"01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f\": rpc error: code = NotFound desc = could not find container \"01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f\": container with ID starting with 01b840508ff608b3900ff6133af677c9b41321cfebee4dccf4d27a541ad32e7f not found: ID does not exist"
Jul 02 13:56:42.584748 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.584725    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"]
Jul 02 13:56:42.591325 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:42.591307    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-59c85f56c9-2z4kl"]
Jul 02 13:56:43.351906 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:43.351878    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="824b6be7-7681-43c0-9268-95b4cfca312e" path="/var/lib/kubelet/pods/824b6be7-7681-43c0-9268-95b4cfca312e/volumes"
Jul 02 13:56:49.443842 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:49.443807    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 13:56:59.445355 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:56:59.445325    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 13:57:11.906739 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:11.906704    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"]
Jul 02 13:57:11.907099 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:11.907047    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824b6be7-7681-43c0-9268-95b4cfca312e" containerName="model-chainer"
Jul 02 13:57:11.907099 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:11.907057    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="824b6be7-7681-43c0-9268-95b4cfca312e" containerName="model-chainer"
Jul 02 13:57:11.907196 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:11.907117    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="824b6be7-7681-43c0-9268-95b4cfca312e" containerName="model-chainer"
Jul 02 13:57:11.909744 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:11.909726    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 13:57:11.912814 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:11.912793    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-88827-serving-cert\""
Jul 02 13:57:11.912901 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:11.912850    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-88827-kube-rbac-proxy-sar-config\""
Jul 02 13:57:11.924415 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:11.924394    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"]
Jul 02 13:57:12.005011 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:12.004973    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74f0795-f1b7-4926-9b67-d7c3396b3f00-openshift-service-ca-bundle\") pod \"sequence-graph-88827-56d8c9dfbf-5ll78\" (UID: \"a74f0795-f1b7-4926-9b67-d7c3396b3f00\") " pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 13:57:12.005011 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:12.005016    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a74f0795-f1b7-4926-9b67-d7c3396b3f00-proxy-tls\") pod \"sequence-graph-88827-56d8c9dfbf-5ll78\" (UID: \"a74f0795-f1b7-4926-9b67-d7c3396b3f00\") " pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 13:57:12.105915 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:12.105877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74f0795-f1b7-4926-9b67-d7c3396b3f00-openshift-service-ca-bundle\") pod \"sequence-graph-88827-56d8c9dfbf-5ll78\" (UID: \"a74f0795-f1b7-4926-9b67-d7c3396b3f00\") " pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 13:57:12.106067 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:12.105924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a74f0795-f1b7-4926-9b67-d7c3396b3f00-proxy-tls\") pod \"sequence-graph-88827-56d8c9dfbf-5ll78\" (UID: \"a74f0795-f1b7-4926-9b67-d7c3396b3f00\") " pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 13:57:12.106121 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:57:12.106102    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-88827-serving-cert: secret "sequence-graph-88827-serving-cert" not found
Jul 02 13:57:12.106208 ip-10-0-138-214 kubenswrapper[2574]: E0702 13:57:12.106198    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a74f0795-f1b7-4926-9b67-d7c3396b3f00-proxy-tls podName:a74f0795-f1b7-4926-9b67-d7c3396b3f00 nodeName:}" failed. No retries permitted until 2026-07-02 13:57:12.606182513 +0000 UTC m=+769.801806001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a74f0795-f1b7-4926-9b67-d7c3396b3f00-proxy-tls") pod "sequence-graph-88827-56d8c9dfbf-5ll78" (UID: "a74f0795-f1b7-4926-9b67-d7c3396b3f00") : secret "sequence-graph-88827-serving-cert" not found
Jul 02 13:57:12.106545 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:12.106526    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74f0795-f1b7-4926-9b67-d7c3396b3f00-openshift-service-ca-bundle\") pod \"sequence-graph-88827-56d8c9dfbf-5ll78\" (UID: \"a74f0795-f1b7-4926-9b67-d7c3396b3f00\") " pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 13:57:12.611589 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:12.611554    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a74f0795-f1b7-4926-9b67-d7c3396b3f00-proxy-tls\") pod \"sequence-graph-88827-56d8c9dfbf-5ll78\" (UID: \"a74f0795-f1b7-4926-9b67-d7c3396b3f00\") " pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 13:57:12.613851 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:12.613822    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a74f0795-f1b7-4926-9b67-d7c3396b3f00-proxy-tls\") pod \"sequence-graph-88827-56d8c9dfbf-5ll78\" (UID: \"a74f0795-f1b7-4926-9b67-d7c3396b3f00\") " pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 13:57:12.819462 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:12.819431    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 13:57:12.946280 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:12.946194    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"]
Jul 02 13:57:12.948621 ip-10-0-138-214 kubenswrapper[2574]: W0702 13:57:12.948592    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74f0795_f1b7_4926_9b67_d7c3396b3f00.slice/crio-a7cd4b469d4abb32087401daddca434c578b01ba714a1e2fee1d57d7189f117b WatchSource:0}: Error finding container a7cd4b469d4abb32087401daddca434c578b01ba714a1e2fee1d57d7189f117b: Status 404 returned error can't find the container with id a7cd4b469d4abb32087401daddca434c578b01ba714a1e2fee1d57d7189f117b
Jul 02 13:57:13.698391 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:13.698350    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" event={"ID":"a74f0795-f1b7-4926-9b67-d7c3396b3f00","Type":"ContainerStarted","Data":"c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c"}
Jul 02 13:57:13.698570 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:13.698405    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" event={"ID":"a74f0795-f1b7-4926-9b67-d7c3396b3f00","Type":"ContainerStarted","Data":"a7cd4b469d4abb32087401daddca434c578b01ba714a1e2fee1d57d7189f117b"}
Jul 02 13:57:13.698570 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:13.698485    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 13:57:13.716431 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:13.716373    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" podStartSLOduration=2.716356552 podStartE2EDuration="2.716356552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:57:13.716288454 +0000 UTC m=+770.911911961" watchObservedRunningTime="2026-07-02 13:57:13.716356552 +0000 UTC m=+770.911980058"
Jul 02 13:57:19.709278 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:57:19.709251    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 13:59:23.334270 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:59:23.334240    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 13:59:23.335647 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:59:23.335623    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 13:59:23.340428 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:59:23.340406    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 13:59:23.341785 ip-10-0-138-214 kubenswrapper[2574]: I0702 13:59:23.341768    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:04:23.361040 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:23.361013    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:04:23.364074 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:23.364055    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:04:23.367425 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:23.367408    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:04:23.370223 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:23.370208    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:04:30.399482 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.399448    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"]
Jul 02 14:04:30.399859 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.399639    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" podUID="b7e56553-8dc5-47e1-9355-cbf353a81268" containerName="switch-graph-68692" containerID="cri-o://f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756" gracePeriod=30
Jul 02 14:04:30.591553 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.591519    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"]
Jul 02 14:04:30.595362 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.595340    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"]
Jul 02 14:04:30.595503 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.595445    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:30.595601 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.595575    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kserve-container" containerID="cri-o://2f153675dc18a2a23a7da747920b6803f858a4b13925d85b1e80045c31773365" gracePeriod=30
Jul 02 14:04:30.595701 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.595648    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kube-rbac-proxy" containerID="cri-o://2a86d0622af4e7c6154f683c8fd20d18f73207503234fca4754c8dea198fc369" gracePeriod=30
Jul 02 14:04:30.598452 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.598427    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c675b-predictor-serving-cert\""
Jul 02 14:04:30.599025 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.599007    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c675b-kube-rbac-proxy-sar-config\""
Jul 02 14:04:30.605281 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.605255    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"]
Jul 02 14:04:30.695334 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.695293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c675b-predictor-7b8c7b5778-542pz\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") " pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:30.695533 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.695388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rkr\" (UniqueName: \"kubernetes.io/projected/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-api-access-76rkr\") pod \"success-200-isvc-c675b-predictor-7b8c7b5778-542pz\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") " pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:30.695533 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.695437    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cecfa45c-56ff-4bf4-a639-530b29bb9418-proxy-tls\") pod \"success-200-isvc-c675b-predictor-7b8c7b5778-542pz\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") " pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:30.796190 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.796154    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c675b-predictor-7b8c7b5778-542pz\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") " pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:30.796360 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.796209    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rkr\" (UniqueName: \"kubernetes.io/projected/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-api-access-76rkr\") pod \"success-200-isvc-c675b-predictor-7b8c7b5778-542pz\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") " pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:30.796360 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.796235    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cecfa45c-56ff-4bf4-a639-530b29bb9418-proxy-tls\") pod \"success-200-isvc-c675b-predictor-7b8c7b5778-542pz\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") " pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:30.796360 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:04:30.796348    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-c675b-predictor-serving-cert: secret "success-200-isvc-c675b-predictor-serving-cert" not found
Jul 02 14:04:30.796478 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:04:30.796398    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cecfa45c-56ff-4bf4-a639-530b29bb9418-proxy-tls podName:cecfa45c-56ff-4bf4-a639-530b29bb9418 nodeName:}" failed. No retries permitted until 2026-07-02 14:04:31.296383434 +0000 UTC m=+1208.492006918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cecfa45c-56ff-4bf4-a639-530b29bb9418-proxy-tls") pod "success-200-isvc-c675b-predictor-7b8c7b5778-542pz" (UID: "cecfa45c-56ff-4bf4-a639-530b29bb9418") : secret "success-200-isvc-c675b-predictor-serving-cert" not found
Jul 02 14:04:30.796869 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.796850    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c675b-predictor-7b8c7b5778-542pz\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") " pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:30.805882 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:30.805857    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rkr\" (UniqueName: \"kubernetes.io/projected/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-api-access-76rkr\") pod \"success-200-isvc-c675b-predictor-7b8c7b5778-542pz\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") " pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:31.300099 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:31.300066    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cecfa45c-56ff-4bf4-a639-530b29bb9418-proxy-tls\") pod \"success-200-isvc-c675b-predictor-7b8c7b5778-542pz\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") " pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:31.302530 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:31.302500    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cecfa45c-56ff-4bf4-a639-530b29bb9418-proxy-tls\") pod \"success-200-isvc-c675b-predictor-7b8c7b5778-542pz\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") " pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:31.518548 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:31.518509    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:31.625869 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:31.625832    2574 generic.go:356] "Generic (PLEG): container finished" podID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerID="2a86d0622af4e7c6154f683c8fd20d18f73207503234fca4754c8dea198fc369" exitCode=2
Jul 02 14:04:31.626046 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:31.625912    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" event={"ID":"0dd64166-d6c9-4373-ac9b-b5d692fd6f62","Type":"ContainerDied","Data":"2a86d0622af4e7c6154f683c8fd20d18f73207503234fca4754c8dea198fc369"}
Jul 02 14:04:31.652484 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:31.652460    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"]
Jul 02 14:04:31.655086 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:04:31.655043    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecfa45c_56ff_4bf4_a639_530b29bb9418.slice/crio-8a7368a0fb05427b5b4587f33266a48f8b0026825906726a90347fc900300d5e WatchSource:0}: Error finding container 8a7368a0fb05427b5b4587f33266a48f8b0026825906726a90347fc900300d5e: Status 404 returned error can't find the container with id 8a7368a0fb05427b5b4587f33266a48f8b0026825906726a90347fc900300d5e
Jul 02 14:04:32.633800 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:32.633766    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" event={"ID":"cecfa45c-56ff-4bf4-a639-530b29bb9418","Type":"ContainerStarted","Data":"0580de691b5f9a6ba7024c66403ebeeea79b3ae6c4329ba8f4112f7faef207ce"}
Jul 02 14:04:32.634320 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:32.633851    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" event={"ID":"cecfa45c-56ff-4bf4-a639-530b29bb9418","Type":"ContainerStarted","Data":"fdbae4c594febed9712ccaf5067367f9058071ad483d068a9c0b876ff614f8e2"}
Jul 02 14:04:32.634320 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:32.633886    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:32.634320 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:32.633914    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" event={"ID":"cecfa45c-56ff-4bf4-a639-530b29bb9418","Type":"ContainerStarted","Data":"8a7368a0fb05427b5b4587f33266a48f8b0026825906726a90347fc900300d5e"}
Jul 02 14:04:32.634320 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:32.633951    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:32.635340 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:32.635311    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 14:04:32.655395 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:32.655348    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" podStartSLOduration=2.655332031 podStartE2EDuration="2.655332031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:04:32.654097547 +0000 UTC m=+1209.849721054" watchObservedRunningTime="2026-07-02 14:04:32.655332031 +0000 UTC m=+1209.850955539"
Jul 02 14:04:33.177458 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.177421    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 02 14:04:33.182772 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.182746    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 14:04:33.462257 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.462175    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" podUID="b7e56553-8dc5-47e1-9355-cbf353a81268" containerName="switch-graph-68692" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:04:33.639654 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.639618    2574 generic.go:356] "Generic (PLEG): container finished" podID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerID="2f153675dc18a2a23a7da747920b6803f858a4b13925d85b1e80045c31773365" exitCode=0
Jul 02 14:04:33.640082 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.639695    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" event={"ID":"0dd64166-d6c9-4373-ac9b-b5d692fd6f62","Type":"ContainerDied","Data":"2f153675dc18a2a23a7da747920b6803f858a4b13925d85b1e80045c31773365"}
Jul 02 14:04:33.640082 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.639973    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 14:04:33.752904 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.752872    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 14:04:33.922902 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.922871    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c86d8\" (UniqueName: \"kubernetes.io/projected/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-api-access-c86d8\") pod \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") "
Jul 02 14:04:33.923066 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.922933    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-proxy-tls\") pod \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") "
Jul 02 14:04:33.923066 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.922959    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-rbac-proxy-sar-config\") pod \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\" (UID: \"0dd64166-d6c9-4373-ac9b-b5d692fd6f62\") "
Jul 02 14:04:33.923375 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.923346    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0dd64166-d6c9-4373-ac9b-b5d692fd6f62" (UID: "0dd64166-d6c9-4373-ac9b-b5d692fd6f62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:04:33.924941 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.924918    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0dd64166-d6c9-4373-ac9b-b5d692fd6f62" (UID: "0dd64166-d6c9-4373-ac9b-b5d692fd6f62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:04:33.925024 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:33.924947    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-api-access-c86d8" (OuterVolumeSpecName: "kube-api-access-c86d8") pod "0dd64166-d6c9-4373-ac9b-b5d692fd6f62" (UID: "0dd64166-d6c9-4373-ac9b-b5d692fd6f62"). InnerVolumeSpecName "kube-api-access-c86d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:04:34.023542 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:34.023481    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c86d8\" (UniqueName: \"kubernetes.io/projected/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-api-access-c86d8\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:04:34.023542 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:34.023506    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:04:34.023542 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:34.023515    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dd64166-d6c9-4373-ac9b-b5d692fd6f62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:04:34.645798 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:34.645770    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"
Jul 02 14:04:34.645798 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:34.645786    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m" event={"ID":"0dd64166-d6c9-4373-ac9b-b5d692fd6f62","Type":"ContainerDied","Data":"c86b350b1746c40e3ac2cb4ff6d8deb67bd9a5d7e11670b235ec9135a73cc4d0"}
Jul 02 14:04:34.646317 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:34.645859    2574 scope.go:117] "RemoveContainer" containerID="2a86d0622af4e7c6154f683c8fd20d18f73207503234fca4754c8dea198fc369"
Jul 02 14:04:34.654357 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:34.654343    2574 scope.go:117] "RemoveContainer" containerID="2f153675dc18a2a23a7da747920b6803f858a4b13925d85b1e80045c31773365"
Jul 02 14:04:34.668824 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:34.668804    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"]
Jul 02 14:04:34.672466 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:34.672447    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-68692-predictor-98ddcc858-6r58m"]
Jul 02 14:04:35.350562 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:35.350534    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" path="/var/lib/kubelet/pods/0dd64166-d6c9-4373-ac9b-b5d692fd6f62/volumes"
Jul 02 14:04:38.462329 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:38.462292    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" podUID="b7e56553-8dc5-47e1-9355-cbf353a81268" containerName="switch-graph-68692" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:04:38.644821 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:38.644791    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:04:38.645235 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:38.645208    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 14:04:43.462077 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:43.462041    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" podUID="b7e56553-8dc5-47e1-9355-cbf353a81268" containerName="switch-graph-68692" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:04:43.462434 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:43.462201    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 14:04:48.462044 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:48.462001    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" podUID="b7e56553-8dc5-47e1-9355-cbf353a81268" containerName="switch-graph-68692" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:04:48.645996 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:48.645958    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 14:04:53.461640 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:53.461600    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" podUID="b7e56553-8dc5-47e1-9355-cbf353a81268" containerName="switch-graph-68692" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:04:58.461815 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:58.461776    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" podUID="b7e56553-8dc5-47e1-9355-cbf353a81268" containerName="switch-graph-68692" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:04:58.646176 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:04:58.646119    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 14:05:00.551717 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.551694    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 14:05:00.641693 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.641666    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e56553-8dc5-47e1-9355-cbf353a81268-proxy-tls\") pod \"b7e56553-8dc5-47e1-9355-cbf353a81268\" (UID: \"b7e56553-8dc5-47e1-9355-cbf353a81268\") "
Jul 02 14:05:00.641881 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.641731    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e56553-8dc5-47e1-9355-cbf353a81268-openshift-service-ca-bundle\") pod \"b7e56553-8dc5-47e1-9355-cbf353a81268\" (UID: \"b7e56553-8dc5-47e1-9355-cbf353a81268\") "
Jul 02 14:05:00.642082 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.642057    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e56553-8dc5-47e1-9355-cbf353a81268-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b7e56553-8dc5-47e1-9355-cbf353a81268" (UID: "b7e56553-8dc5-47e1-9355-cbf353a81268"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:05:00.643590 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.643564    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e56553-8dc5-47e1-9355-cbf353a81268-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7e56553-8dc5-47e1-9355-cbf353a81268" (UID: "b7e56553-8dc5-47e1-9355-cbf353a81268"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:05:00.743234 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.743209    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7e56553-8dc5-47e1-9355-cbf353a81268-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:05:00.743234 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.743234    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e56553-8dc5-47e1-9355-cbf353a81268-openshift-service-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:05:00.764364 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.764328    2574 generic.go:356] "Generic (PLEG): container finished" podID="b7e56553-8dc5-47e1-9355-cbf353a81268" containerID="f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756" exitCode=0
Jul 02 14:05:00.764492 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.764376    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"
Jul 02 14:05:00.764492 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.764391    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" event={"ID":"b7e56553-8dc5-47e1-9355-cbf353a81268","Type":"ContainerDied","Data":"f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756"}
Jul 02 14:05:00.764492 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.764447    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8" event={"ID":"b7e56553-8dc5-47e1-9355-cbf353a81268","Type":"ContainerDied","Data":"76d8422ed9c26887f494b7a90d409823fcc7e1f60fa1d3eb7d9ba0686136ba5f"}
Jul 02 14:05:00.764492 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.764470    2574 scope.go:117] "RemoveContainer" containerID="f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756"
Jul 02 14:05:00.773332 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.773316    2574 scope.go:117] "RemoveContainer" containerID="f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756"
Jul 02 14:05:00.773583 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:05:00.773564    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756\": container with ID starting with f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756 not found: ID does not exist" containerID="f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756"
Jul 02 14:05:00.773648 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.773595    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756"} err="failed to get container status \"f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756\": rpc error: code = NotFound desc = could not find container \"f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756\": container with ID starting with f62801b67a7da3013961390bf00b2867bd8e649e663b526c4dc61cf5b08fd756 not found: ID does not exist"
Jul 02 14:05:00.785731 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.785705    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"]
Jul 02 14:05:00.789537 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:00.789518    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-68692-78776d6ccd-fsth8"]
Jul 02 14:05:01.350355 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:01.350325    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e56553-8dc5-47e1-9355-cbf353a81268" path="/var/lib/kubelet/pods/b7e56553-8dc5-47e1-9355-cbf353a81268/volumes"
Jul 02 14:05:08.645811 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:08.645771    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 02 14:05:18.646828 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:18.646755    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:05:26.610594 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.610556    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"]
Jul 02 14:05:26.611062 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.610811    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" podUID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" containerName="sequence-graph-88827" containerID="cri-o://c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c" gracePeriod=30
Jul 02 14:05:26.763189 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.763157    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"]
Jul 02 14:05:26.763447 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.763401    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kserve-container" containerID="cri-o://eae4ed013b8b5787382bdd4e15b8e12a2d3383258d5ae9a07cf9148f1064d6a4" gracePeriod=30
Jul 02 14:05:26.763631 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.763491    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kube-rbac-proxy" containerID="cri-o://0875218296ed58854097ccd311e7ab88c57a76eba33112c40daf1fdecf563018" gracePeriod=30
Jul 02 14:05:26.775186 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.775107    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"]
Jul 02 14:05:26.776030 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.776004    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kserve-container"
Jul 02 14:05:26.776151 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.776033    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kserve-container"
Jul 02 14:05:26.776151 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.776091    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e56553-8dc5-47e1-9355-cbf353a81268" containerName="switch-graph-68692"
Jul 02 14:05:26.776151 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.776102    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e56553-8dc5-47e1-9355-cbf353a81268" containerName="switch-graph-68692"
Jul 02 14:05:26.776319 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.776190    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kube-rbac-proxy"
Jul 02 14:05:26.776319 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.776201    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kube-rbac-proxy"
Jul 02 14:05:26.776418 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.776368    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e56553-8dc5-47e1-9355-cbf353a81268" containerName="switch-graph-68692"
Jul 02 14:05:26.776418 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.776395    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kube-rbac-proxy"
Jul 02 14:05:26.776418 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.776406    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd64166-d6c9-4373-ac9b-b5d692fd6f62" containerName="kserve-container"
Jul 02 14:05:26.781951 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.781927    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:26.784459 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.784435    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c559a-predictor-serving-cert\""
Jul 02 14:05:26.784562 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.784536    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c559a-kube-rbac-proxy-sar-config\""
Jul 02 14:05:26.788653 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.788632    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"]
Jul 02 14:05:26.859839 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.859797    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/915650ad-3f44-487c-802d-23748fce727c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c559a-predictor-665765c694-bj667\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") " pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:26.860077 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.860055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915650ad-3f44-487c-802d-23748fce727c-proxy-tls\") pod \"success-200-isvc-c559a-predictor-665765c694-bj667\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") " pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:26.860166 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.860112    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nk5h\" (UniqueName: \"kubernetes.io/projected/915650ad-3f44-487c-802d-23748fce727c-kube-api-access-8nk5h\") pod \"success-200-isvc-c559a-predictor-665765c694-bj667\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") " pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:26.879072 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.879008    2574 generic.go:356] "Generic (PLEG): container finished" podID="71af5689-176e-4d90-8d89-d5a70a978f78" containerID="0875218296ed58854097ccd311e7ab88c57a76eba33112c40daf1fdecf563018" exitCode=2
Jul 02 14:05:26.879072 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.879056    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" event={"ID":"71af5689-176e-4d90-8d89-d5a70a978f78","Type":"ContainerDied","Data":"0875218296ed58854097ccd311e7ab88c57a76eba33112c40daf1fdecf563018"}
Jul 02 14:05:26.961363 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.961335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915650ad-3f44-487c-802d-23748fce727c-proxy-tls\") pod \"success-200-isvc-c559a-predictor-665765c694-bj667\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") " pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:26.961524 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.961379    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nk5h\" (UniqueName: \"kubernetes.io/projected/915650ad-3f44-487c-802d-23748fce727c-kube-api-access-8nk5h\") pod \"success-200-isvc-c559a-predictor-665765c694-bj667\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") " pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:26.961524 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.961422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/915650ad-3f44-487c-802d-23748fce727c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c559a-predictor-665765c694-bj667\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") " pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:26.961524 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:05:26.961486    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-c559a-predictor-serving-cert: secret "success-200-isvc-c559a-predictor-serving-cert" not found
Jul 02 14:05:26.961660 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:05:26.961551    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/915650ad-3f44-487c-802d-23748fce727c-proxy-tls podName:915650ad-3f44-487c-802d-23748fce727c nodeName:}" failed. No retries permitted until 2026-07-02 14:05:27.461531003 +0000 UTC m=+1264.657154503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/915650ad-3f44-487c-802d-23748fce727c-proxy-tls") pod "success-200-isvc-c559a-predictor-665765c694-bj667" (UID: "915650ad-3f44-487c-802d-23748fce727c") : secret "success-200-isvc-c559a-predictor-serving-cert" not found
Jul 02 14:05:26.962022 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.962005    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/915650ad-3f44-487c-802d-23748fce727c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c559a-predictor-665765c694-bj667\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") " pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:26.972011 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:26.971984    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nk5h\" (UniqueName: \"kubernetes.io/projected/915650ad-3f44-487c-802d-23748fce727c-kube-api-access-8nk5h\") pod \"success-200-isvc-c559a-predictor-665765c694-bj667\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") " pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:27.466182 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:27.466150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915650ad-3f44-487c-802d-23748fce727c-proxy-tls\") pod \"success-200-isvc-c559a-predictor-665765c694-bj667\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") " pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:27.468529 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:27.468507    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915650ad-3f44-487c-802d-23748fce727c-proxy-tls\") pod \"success-200-isvc-c559a-predictor-665765c694-bj667\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") " pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:27.693321 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:27.693277    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:27.829081 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:27.829051    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"]
Jul 02 14:05:27.830633 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:05:27.830603    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915650ad_3f44_487c_802d_23748fce727c.slice/crio-8a6cf040496fe2f30574fad133da26e835d55c71ef2b69e7dafed28c0b5ae225 WatchSource:0}: Error finding container 8a6cf040496fe2f30574fad133da26e835d55c71ef2b69e7dafed28c0b5ae225: Status 404 returned error can't find the container with id 8a6cf040496fe2f30574fad133da26e835d55c71ef2b69e7dafed28c0b5ae225
Jul 02 14:05:27.885676 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:27.885650    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" event={"ID":"915650ad-3f44-487c-802d-23748fce727c","Type":"ContainerStarted","Data":"8a6cf040496fe2f30574fad133da26e835d55c71ef2b69e7dafed28c0b5ae225"}
Jul 02 14:05:28.892456 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:28.892423    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" event={"ID":"915650ad-3f44-487c-802d-23748fce727c","Type":"ContainerStarted","Data":"87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d"}
Jul 02 14:05:28.892848 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:28.892479    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" event={"ID":"915650ad-3f44-487c-802d-23748fce727c","Type":"ContainerStarted","Data":"819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd"}
Jul 02 14:05:28.892848 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:28.892505    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:28.912254 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:28.912212    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" podStartSLOduration=2.91219621 podStartE2EDuration="2.91219621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:05:28.910666257 +0000 UTC m=+1266.106289768" watchObservedRunningTime="2026-07-02 14:05:28.91219621 +0000 UTC m=+1266.107819717"
Jul 02 14:05:29.439193 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:29.439155    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 02 14:05:29.444485 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:29.444462    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 02 14:05:29.707304 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:29.707225    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" podUID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" containerName="sequence-graph-88827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:05:29.899021 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:29.898993    2574 generic.go:356] "Generic (PLEG): container finished" podID="71af5689-176e-4d90-8d89-d5a70a978f78" containerID="eae4ed013b8b5787382bdd4e15b8e12a2d3383258d5ae9a07cf9148f1064d6a4" exitCode=0
Jul 02 14:05:29.899349 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:29.899064    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" event={"ID":"71af5689-176e-4d90-8d89-d5a70a978f78","Type":"ContainerDied","Data":"eae4ed013b8b5787382bdd4e15b8e12a2d3383258d5ae9a07cf9148f1064d6a4"}
Jul 02 14:05:29.899349 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:29.899125    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz" event={"ID":"71af5689-176e-4d90-8d89-d5a70a978f78","Type":"ContainerDied","Data":"0aacad459935d4071ea9ef1e17352b971f0b404c53a921648b0ff5d9820e6922"}
Jul 02 14:05:29.899349 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:29.899172    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aacad459935d4071ea9ef1e17352b971f0b404c53a921648b0ff5d9820e6922"
Jul 02 14:05:29.899453 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:29.899351    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:29.900778 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:29.900759    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 14:05:29.908043 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:29.908026    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 14:05:30.087801 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.087721    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dt2x\" (UniqueName: \"kubernetes.io/projected/71af5689-176e-4d90-8d89-d5a70a978f78-kube-api-access-6dt2x\") pod \"71af5689-176e-4d90-8d89-d5a70a978f78\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") "
Jul 02 14:05:30.087801 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.087762    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71af5689-176e-4d90-8d89-d5a70a978f78-kube-rbac-proxy-sar-config\") pod \"71af5689-176e-4d90-8d89-d5a70a978f78\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") "
Jul 02 14:05:30.087801 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.087794    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71af5689-176e-4d90-8d89-d5a70a978f78-proxy-tls\") pod \"71af5689-176e-4d90-8d89-d5a70a978f78\" (UID: \"71af5689-176e-4d90-8d89-d5a70a978f78\") "
Jul 02 14:05:30.088214 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.088189    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71af5689-176e-4d90-8d89-d5a70a978f78-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71af5689-176e-4d90-8d89-d5a70a978f78" (UID: "71af5689-176e-4d90-8d89-d5a70a978f78"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:05:30.089888 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.089857    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71af5689-176e-4d90-8d89-d5a70a978f78-kube-api-access-6dt2x" (OuterVolumeSpecName: "kube-api-access-6dt2x") pod "71af5689-176e-4d90-8d89-d5a70a978f78" (UID: "71af5689-176e-4d90-8d89-d5a70a978f78"). InnerVolumeSpecName "kube-api-access-6dt2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:05:30.089888 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.089856    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71af5689-176e-4d90-8d89-d5a70a978f78-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71af5689-176e-4d90-8d89-d5a70a978f78" (UID: "71af5689-176e-4d90-8d89-d5a70a978f78"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:05:30.189146 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.189104    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dt2x\" (UniqueName: \"kubernetes.io/projected/71af5689-176e-4d90-8d89-d5a70a978f78-kube-api-access-6dt2x\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:05:30.189146 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.189129    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71af5689-176e-4d90-8d89-d5a70a978f78-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:05:30.189315 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.189160    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71af5689-176e-4d90-8d89-d5a70a978f78-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:05:30.618853 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.618823    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"]
Jul 02 14:05:30.619192 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.619177    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kube-rbac-proxy"
Jul 02 14:05:30.619242 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.619193    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kube-rbac-proxy"
Jul 02 14:05:30.619242 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.619208    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kserve-container"
Jul 02 14:05:30.619242 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.619214    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kserve-container"
Jul 02 14:05:30.619336 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.619286    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kserve-container"
Jul 02 14:05:30.619336 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.619296    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" containerName="kube-rbac-proxy"
Jul 02 14:05:30.622314 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.622294    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:05:30.624689 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.624672    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c675b-serving-cert\""
Jul 02 14:05:30.624776 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.624705    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c675b-kube-rbac-proxy-sar-config\""
Jul 02 14:05:30.630624 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.630601    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"]
Jul 02 14:05:30.793804 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.793778    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/816c127e-16bd-4132-881c-3199fe1e0907-proxy-tls\") pod \"ensemble-graph-c675b-768bbd884f-vxx9q\" (UID: \"816c127e-16bd-4132-881c-3199fe1e0907\") " pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:05:30.793964 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.793819    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816c127e-16bd-4132-881c-3199fe1e0907-openshift-service-ca-bundle\") pod \"ensemble-graph-c675b-768bbd884f-vxx9q\" (UID: \"816c127e-16bd-4132-881c-3199fe1e0907\") " pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:05:30.895267 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.895181    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816c127e-16bd-4132-881c-3199fe1e0907-openshift-service-ca-bundle\") pod \"ensemble-graph-c675b-768bbd884f-vxx9q\" (UID: \"816c127e-16bd-4132-881c-3199fe1e0907\") " pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:05:30.895410 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.895279    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/816c127e-16bd-4132-881c-3199fe1e0907-proxy-tls\") pod \"ensemble-graph-c675b-768bbd884f-vxx9q\" (UID: \"816c127e-16bd-4132-881c-3199fe1e0907\") " pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:05:30.895938 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.895916    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816c127e-16bd-4132-881c-3199fe1e0907-openshift-service-ca-bundle\") pod \"ensemble-graph-c675b-768bbd884f-vxx9q\" (UID: \"816c127e-16bd-4132-881c-3199fe1e0907\") " pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:05:30.897790 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.897762    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/816c127e-16bd-4132-881c-3199fe1e0907-proxy-tls\") pod \"ensemble-graph-c675b-768bbd884f-vxx9q\" (UID: \"816c127e-16bd-4132-881c-3199fe1e0907\") " pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:05:30.903624 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.903492    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"
Jul 02 14:05:30.904031 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.903630    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 14:05:30.929047 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.929014    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"]
Jul 02 14:05:30.932932 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.932909    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88827-predictor-68bb654446-xn6vz"]
Jul 02 14:05:30.933053 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:30.933001    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:05:31.084981 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:31.084950    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"]
Jul 02 14:05:31.087965 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:05:31.087941    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816c127e_16bd_4132_881c_3199fe1e0907.slice/crio-e11b40eff0f10228b7413ba2b1ee0e5fc4549cbf5bb04d8486fab2fea639bf8e WatchSource:0}: Error finding container e11b40eff0f10228b7413ba2b1ee0e5fc4549cbf5bb04d8486fab2fea639bf8e: Status 404 returned error can't find the container with id e11b40eff0f10228b7413ba2b1ee0e5fc4549cbf5bb04d8486fab2fea639bf8e
Jul 02 14:05:31.352312 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:31.352283    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71af5689-176e-4d90-8d89-d5a70a978f78" path="/var/lib/kubelet/pods/71af5689-176e-4d90-8d89-d5a70a978f78/volumes"
Jul 02 14:05:31.909592 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:31.909554    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" event={"ID":"816c127e-16bd-4132-881c-3199fe1e0907","Type":"ContainerStarted","Data":"8dbe0a77e8c05b4b569b1c45719f7de4c476b9d44f62503140293ea712434fe9"}
Jul 02 14:05:31.910098 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:31.909608    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" event={"ID":"816c127e-16bd-4132-881c-3199fe1e0907","Type":"ContainerStarted","Data":"e11b40eff0f10228b7413ba2b1ee0e5fc4549cbf5bb04d8486fab2fea639bf8e"}
Jul 02 14:05:31.910098 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:31.909681    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:05:31.929564 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:31.929515    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" podStartSLOduration=1.929498197 podStartE2EDuration="1.929498197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:05:31.928502915 +0000 UTC m=+1269.124126422" watchObservedRunningTime="2026-07-02 14:05:31.929498197 +0000 UTC m=+1269.125121701"
Jul 02 14:05:34.706808 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:34.706769    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" podUID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" containerName="sequence-graph-88827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:05:35.909108 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:35.909083    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:05:35.909653 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:35.909626    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 14:05:37.920365 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:37.920338    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:05:39.707424 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:39.707389    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" podUID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" containerName="sequence-graph-88827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:05:39.707793 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:39.707498    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 14:05:40.709151 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.709103    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"]
Jul 02 14:05:40.709578 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.709485    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" podUID="816c127e-16bd-4132-881c-3199fe1e0907" containerName="ensemble-graph-c675b" containerID="cri-o://8dbe0a77e8c05b4b569b1c45719f7de4c476b9d44f62503140293ea712434fe9" gracePeriod=30
Jul 02 14:05:40.843056 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.843030    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"]
Jul 02 14:05:40.843288 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.843263    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kserve-container" containerID="cri-o://fdbae4c594febed9712ccaf5067367f9058071ad483d068a9c0b876ff614f8e2" gracePeriod=30
Jul 02 14:05:40.843342 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.843304    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kube-rbac-proxy" containerID="cri-o://0580de691b5f9a6ba7024c66403ebeeea79b3ae6c4329ba8f4112f7faef207ce" gracePeriod=30
Jul 02 14:05:40.856511 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.856490    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"]
Jul 02 14:05:40.861476 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.861458    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:40.863923 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.863902    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7b7ab-kube-rbac-proxy-sar-config\""
Jul 02 14:05:40.864009 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.863903    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7b7ab-predictor-serving-cert\""
Jul 02 14:05:40.870508 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.870485    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"]
Jul 02 14:05:40.878247 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.878227    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285570f8-0db8-42d5-8404-1e07570d46a0-proxy-tls\") pod \"success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") " pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:40.878342 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.878279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285570f8-0db8-42d5-8404-1e07570d46a0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") " pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:40.878342 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.878314    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7lpz\" (UniqueName: \"kubernetes.io/projected/285570f8-0db8-42d5-8404-1e07570d46a0-kube-api-access-l7lpz\") pod \"success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") " pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:40.979559 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.979483    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285570f8-0db8-42d5-8404-1e07570d46a0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") " pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:40.979703 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.979534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7lpz\" (UniqueName: \"kubernetes.io/projected/285570f8-0db8-42d5-8404-1e07570d46a0-kube-api-access-l7lpz\") pod \"success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") " pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:40.979703 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.979654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285570f8-0db8-42d5-8404-1e07570d46a0-proxy-tls\") pod \"success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") " pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:40.979818 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:05:40.979800    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-serving-cert: secret "success-200-isvc-7b7ab-predictor-serving-cert" not found
Jul 02 14:05:40.979908 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:05:40.979889    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285570f8-0db8-42d5-8404-1e07570d46a0-proxy-tls podName:285570f8-0db8-42d5-8404-1e07570d46a0 nodeName:}" failed. No retries permitted until 2026-07-02 14:05:41.479857574 +0000 UTC m=+1278.675481064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/285570f8-0db8-42d5-8404-1e07570d46a0-proxy-tls") pod "success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" (UID: "285570f8-0db8-42d5-8404-1e07570d46a0") : secret "success-200-isvc-7b7ab-predictor-serving-cert" not found
Jul 02 14:05:40.980199 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.980181    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285570f8-0db8-42d5-8404-1e07570d46a0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") " pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:40.990498 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:40.990473    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7lpz\" (UniqueName: \"kubernetes.io/projected/285570f8-0db8-42d5-8404-1e07570d46a0-kube-api-access-l7lpz\") pod \"success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") " pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:41.484637 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:41.484583    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285570f8-0db8-42d5-8404-1e07570d46a0-proxy-tls\") pod \"success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") " pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:41.487112 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:41.487087    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285570f8-0db8-42d5-8404-1e07570d46a0-proxy-tls\") pod \"success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") " pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:41.772310 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:41.772225    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:41.901389 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:41.901345    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"]
Jul 02 14:05:41.903407 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:05:41.903381    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285570f8_0db8_42d5_8404_1e07570d46a0.slice/crio-f668a3af5eaeb9f69be8beef188868c3a4bc942ef93b71d7c4c1f82f9b8edb02 WatchSource:0}: Error finding container f668a3af5eaeb9f69be8beef188868c3a4bc942ef93b71d7c4c1f82f9b8edb02: Status 404 returned error can't find the container with id f668a3af5eaeb9f69be8beef188868c3a4bc942ef93b71d7c4c1f82f9b8edb02
Jul 02 14:05:41.957366 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:41.957340    2574 generic.go:356] "Generic (PLEG): container finished" podID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerID="0580de691b5f9a6ba7024c66403ebeeea79b3ae6c4329ba8f4112f7faef207ce" exitCode=2
Jul 02 14:05:41.957466 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:41.957409    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" event={"ID":"cecfa45c-56ff-4bf4-a639-530b29bb9418","Type":"ContainerDied","Data":"0580de691b5f9a6ba7024c66403ebeeea79b3ae6c4329ba8f4112f7faef207ce"}
Jul 02 14:05:41.958504 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:41.958483    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" event={"ID":"285570f8-0db8-42d5-8404-1e07570d46a0","Type":"ContainerStarted","Data":"f668a3af5eaeb9f69be8beef188868c3a4bc942ef93b71d7c4c1f82f9b8edb02"}
Jul 02 14:05:42.918351 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:42.918313    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" podUID="816c127e-16bd-4132-881c-3199fe1e0907" containerName="ensemble-graph-c675b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:05:42.964956 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:42.964920    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" event={"ID":"285570f8-0db8-42d5-8404-1e07570d46a0","Type":"ContainerStarted","Data":"dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4"}
Jul 02 14:05:42.965111 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:42.964983    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" event={"ID":"285570f8-0db8-42d5-8404-1e07570d46a0","Type":"ContainerStarted","Data":"c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f"}
Jul 02 14:05:42.965111 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:42.965011    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:42.965233 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:42.965163    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:42.966728 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:42.966702    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 14:05:42.983810 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:42.983761    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" podStartSLOduration=2.983746053 podStartE2EDuration="2.983746053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:05:42.982635447 +0000 UTC m=+1280.178258992" watchObservedRunningTime="2026-07-02 14:05:42.983746053 +0000 UTC m=+1280.179369561"
Jul 02 14:05:43.640250 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:43.640206    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 02 14:05:43.970753 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:43.970724    2574 generic.go:356] "Generic (PLEG): container finished" podID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerID="fdbae4c594febed9712ccaf5067367f9058071ad483d068a9c0b876ff614f8e2" exitCode=0
Jul 02 14:05:43.971095 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:43.970801    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" event={"ID":"cecfa45c-56ff-4bf4-a639-530b29bb9418","Type":"ContainerDied","Data":"fdbae4c594febed9712ccaf5067367f9058071ad483d068a9c0b876ff614f8e2"}
Jul 02 14:05:43.971238 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:43.971208    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 14:05:44.087081 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.087060    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:05:44.105488 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.105467    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cecfa45c-56ff-4bf4-a639-530b29bb9418-proxy-tls\") pod \"cecfa45c-56ff-4bf4-a639-530b29bb9418\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") "
Jul 02 14:05:44.105561 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.105506    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76rkr\" (UniqueName: \"kubernetes.io/projected/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-api-access-76rkr\") pod \"cecfa45c-56ff-4bf4-a639-530b29bb9418\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") "
Jul 02 14:05:44.105561 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.105537    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-rbac-proxy-sar-config\") pod \"cecfa45c-56ff-4bf4-a639-530b29bb9418\" (UID: \"cecfa45c-56ff-4bf4-a639-530b29bb9418\") "
Jul 02 14:05:44.105912 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.105892    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cecfa45c-56ff-4bf4-a639-530b29bb9418" (UID: "cecfa45c-56ff-4bf4-a639-530b29bb9418"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:05:44.107516 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.107494    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cecfa45c-56ff-4bf4-a639-530b29bb9418-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cecfa45c-56ff-4bf4-a639-530b29bb9418" (UID: "cecfa45c-56ff-4bf4-a639-530b29bb9418"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:05:44.107585 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.107495    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-api-access-76rkr" (OuterVolumeSpecName: "kube-api-access-76rkr") pod "cecfa45c-56ff-4bf4-a639-530b29bb9418" (UID: "cecfa45c-56ff-4bf4-a639-530b29bb9418"). InnerVolumeSpecName "kube-api-access-76rkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:05:44.206965 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.206895    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cecfa45c-56ff-4bf4-a639-530b29bb9418-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:05:44.206965 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.206919    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76rkr\" (UniqueName: \"kubernetes.io/projected/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-api-access-76rkr\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:05:44.206965 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.206929    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cecfa45c-56ff-4bf4-a639-530b29bb9418-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:05:44.707521 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.707478    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" podUID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" containerName="sequence-graph-88827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:05:44.976783 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.976703    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"
Jul 02 14:05:44.976783 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.976738    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz" event={"ID":"cecfa45c-56ff-4bf4-a639-530b29bb9418","Type":"ContainerDied","Data":"8a7368a0fb05427b5b4587f33266a48f8b0026825906726a90347fc900300d5e"}
Jul 02 14:05:44.977253 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.976812    2574 scope.go:117] "RemoveContainer" containerID="0580de691b5f9a6ba7024c66403ebeeea79b3ae6c4329ba8f4112f7faef207ce"
Jul 02 14:05:44.985820 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.985804    2574 scope.go:117] "RemoveContainer" containerID="fdbae4c594febed9712ccaf5067367f9058071ad483d068a9c0b876ff614f8e2"
Jul 02 14:05:44.998905 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:44.998882    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"]
Jul 02 14:05:45.005687 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:45.005666    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c675b-predictor-7b8c7b5778-542pz"]
Jul 02 14:05:45.350929 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:45.350858    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" path="/var/lib/kubelet/pods/cecfa45c-56ff-4bf4-a639-530b29bb9418/volumes"
Jul 02 14:05:45.910041 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:45.910000    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 14:05:47.919374 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:47.919337    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" podUID="816c127e-16bd-4132-881c-3199fe1e0907" containerName="ensemble-graph-c675b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:05:48.975812 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:48.975784    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:05:48.976200 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:48.976149    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 14:05:49.706582 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:49.706544    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" podUID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" containerName="sequence-graph-88827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:05:52.918509 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:52.918472    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" podUID="816c127e-16bd-4132-881c-3199fe1e0907" containerName="ensemble-graph-c675b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:05:52.918923 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:52.918571    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:05:54.707080 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:54.707044    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" podUID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" containerName="sequence-graph-88827" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:05:55.909709 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:55.909676    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 14:05:56.763340 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:56.763314    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 14:05:56.812824 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:56.812797    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74f0795-f1b7-4926-9b67-d7c3396b3f00-openshift-service-ca-bundle\") pod \"a74f0795-f1b7-4926-9b67-d7c3396b3f00\" (UID: \"a74f0795-f1b7-4926-9b67-d7c3396b3f00\") "
Jul 02 14:05:56.812988 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:56.812892    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a74f0795-f1b7-4926-9b67-d7c3396b3f00-proxy-tls\") pod \"a74f0795-f1b7-4926-9b67-d7c3396b3f00\" (UID: \"a74f0795-f1b7-4926-9b67-d7c3396b3f00\") "
Jul 02 14:05:56.813232 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:56.813203    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74f0795-f1b7-4926-9b67-d7c3396b3f00-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a74f0795-f1b7-4926-9b67-d7c3396b3f00" (UID: "a74f0795-f1b7-4926-9b67-d7c3396b3f00"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:05:56.814807 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:56.814789    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74f0795-f1b7-4926-9b67-d7c3396b3f00-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a74f0795-f1b7-4926-9b67-d7c3396b3f00" (UID: "a74f0795-f1b7-4926-9b67-d7c3396b3f00"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:05:56.913621 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:56.913598    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74f0795-f1b7-4926-9b67-d7c3396b3f00-openshift-service-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:05:56.913621 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:56.913623    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a74f0795-f1b7-4926-9b67-d7c3396b3f00-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:05:57.036893 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:57.036852    2574 generic.go:356] "Generic (PLEG): container finished" podID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" containerID="c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c" exitCode=0
Jul 02 14:05:57.037051 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:57.036907    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"
Jul 02 14:05:57.037051 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:57.036963    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" event={"ID":"a74f0795-f1b7-4926-9b67-d7c3396b3f00","Type":"ContainerDied","Data":"c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c"}
Jul 02 14:05:57.037051 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:57.037015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78" event={"ID":"a74f0795-f1b7-4926-9b67-d7c3396b3f00","Type":"ContainerDied","Data":"a7cd4b469d4abb32087401daddca434c578b01ba714a1e2fee1d57d7189f117b"}
Jul 02 14:05:57.037051 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:57.037037    2574 scope.go:117] "RemoveContainer" containerID="c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c"
Jul 02 14:05:57.046437 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:57.046335    2574 scope.go:117] "RemoveContainer" containerID="c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c"
Jul 02 14:05:57.046643 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:05:57.046619    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c\": container with ID starting with c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c not found: ID does not exist" containerID="c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c"
Jul 02 14:05:57.046760 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:57.046651    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c"} err="failed to get container status \"c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c\": rpc error: code = NotFound desc = could not find container \"c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c\": container with ID starting with c331c47c33795e60e84652c4662a4e2e57be93b81d546a0408810a7a7daf197c not found: ID does not exist"
Jul 02 14:05:57.059367 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:57.059344    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"]
Jul 02 14:05:57.063238 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:57.063219    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-88827-56d8c9dfbf-5ll78"]
Jul 02 14:05:57.351052 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:57.350970    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" path="/var/lib/kubelet/pods/a74f0795-f1b7-4926-9b67-d7c3396b3f00/volumes"
Jul 02 14:05:57.918448 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:57.918415    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" podUID="816c127e-16bd-4132-881c-3199fe1e0907" containerName="ensemble-graph-c675b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:05:58.977083 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:05:58.977048    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 14:06:02.918973 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:02.918932    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" podUID="816c127e-16bd-4132-881c-3199fe1e0907" containerName="ensemble-graph-c675b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:06:05.910192 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:05.910125    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 14:06:07.918962 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:07.918921    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" podUID="816c127e-16bd-4132-881c-3199fe1e0907" containerName="ensemble-graph-c675b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:06:08.977099 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:08.977062    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 14:06:11.100316 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:11.100229    2574 generic.go:356] "Generic (PLEG): container finished" podID="816c127e-16bd-4132-881c-3199fe1e0907" containerID="8dbe0a77e8c05b4b569b1c45719f7de4c476b9d44f62503140293ea712434fe9" exitCode=0
Jul 02 14:06:11.100316 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:11.100302    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" event={"ID":"816c127e-16bd-4132-881c-3199fe1e0907","Type":"ContainerDied","Data":"8dbe0a77e8c05b4b569b1c45719f7de4c476b9d44f62503140293ea712434fe9"}
Jul 02 14:06:11.359669 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:11.359617    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:06:11.437362 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:11.437327    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/816c127e-16bd-4132-881c-3199fe1e0907-proxy-tls\") pod \"816c127e-16bd-4132-881c-3199fe1e0907\" (UID: \"816c127e-16bd-4132-881c-3199fe1e0907\") "
Jul 02 14:06:11.437362 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:11.437364    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816c127e-16bd-4132-881c-3199fe1e0907-openshift-service-ca-bundle\") pod \"816c127e-16bd-4132-881c-3199fe1e0907\" (UID: \"816c127e-16bd-4132-881c-3199fe1e0907\") "
Jul 02 14:06:11.437749 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:11.437720    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816c127e-16bd-4132-881c-3199fe1e0907-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "816c127e-16bd-4132-881c-3199fe1e0907" (UID: "816c127e-16bd-4132-881c-3199fe1e0907"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:06:11.439351 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:11.439328    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816c127e-16bd-4132-881c-3199fe1e0907-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "816c127e-16bd-4132-881c-3199fe1e0907" (UID: "816c127e-16bd-4132-881c-3199fe1e0907"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:06:11.538895 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:11.538854    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/816c127e-16bd-4132-881c-3199fe1e0907-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:06:11.538895 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:11.538886    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816c127e-16bd-4132-881c-3199fe1e0907-openshift-service-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:06:12.105627 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:12.105595    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"
Jul 02 14:06:12.106093 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:12.105632    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q" event={"ID":"816c127e-16bd-4132-881c-3199fe1e0907","Type":"ContainerDied","Data":"e11b40eff0f10228b7413ba2b1ee0e5fc4549cbf5bb04d8486fab2fea639bf8e"}
Jul 02 14:06:12.106093 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:12.105701    2574 scope.go:117] "RemoveContainer" containerID="8dbe0a77e8c05b4b569b1c45719f7de4c476b9d44f62503140293ea712434fe9"
Jul 02 14:06:12.127729 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:12.127698    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"]
Jul 02 14:06:12.132637 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:12.132610    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c675b-768bbd884f-vxx9q"]
Jul 02 14:06:13.350034 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:13.350000    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="816c127e-16bd-4132-881c-3199fe1e0907" path="/var/lib/kubelet/pods/816c127e-16bd-4132-881c-3199fe1e0907/volumes"
Jul 02 14:06:15.911290 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:15.911263    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:06:18.977164 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:18.977110    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 14:06:23.425084 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:23.425052    2574 scope.go:117] "RemoveContainer" containerID="0875218296ed58854097ccd311e7ab88c57a76eba33112c40daf1fdecf563018"
Jul 02 14:06:23.435844 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:23.435818    2574 scope.go:117] "RemoveContainer" containerID="eae4ed013b8b5787382bdd4e15b8e12a2d3383258d5ae9a07cf9148f1064d6a4"
Jul 02 14:06:26.801828 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.801795    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"]
Jul 02 14:06:26.802241 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802117    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kube-rbac-proxy"
Jul 02 14:06:26.802241 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802127    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kube-rbac-proxy"
Jul 02 14:06:26.802241 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802156    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" containerName="sequence-graph-88827"
Jul 02 14:06:26.802241 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802165    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" containerName="sequence-graph-88827"
Jul 02 14:06:26.802241 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802181    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kserve-container"
Jul 02 14:06:26.802241 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802190    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kserve-container"
Jul 02 14:06:26.802241 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802201    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="816c127e-16bd-4132-881c-3199fe1e0907" containerName="ensemble-graph-c675b"
Jul 02 14:06:26.802241 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802206    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="816c127e-16bd-4132-881c-3199fe1e0907" containerName="ensemble-graph-c675b"
Jul 02 14:06:26.802494 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802260    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kube-rbac-proxy"
Jul 02 14:06:26.802494 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802270    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cecfa45c-56ff-4bf4-a639-530b29bb9418" containerName="kserve-container"
Jul 02 14:06:26.802494 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802277    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74f0795-f1b7-4926-9b67-d7c3396b3f00" containerName="sequence-graph-88827"
Jul 02 14:06:26.802494 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.802283    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="816c127e-16bd-4132-881c-3199fe1e0907" containerName="ensemble-graph-c675b"
Jul 02 14:06:26.806016 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.805996    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:06:26.808484 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.808465    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c559a-kube-rbac-proxy-sar-config\""
Jul 02 14:06:26.808584 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.808468    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-c559a-serving-cert\""
Jul 02 14:06:26.813994 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.813972    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"]
Jul 02 14:06:26.970722 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.970690    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa15671a-0497-4910-81be-a7876a3f9e03-openshift-service-ca-bundle\") pod \"sequence-graph-c559a-57c5445779-r4zfm\" (UID: \"aa15671a-0497-4910-81be-a7876a3f9e03\") " pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:06:26.970928 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:26.970822    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa15671a-0497-4910-81be-a7876a3f9e03-proxy-tls\") pod \"sequence-graph-c559a-57c5445779-r4zfm\" (UID: \"aa15671a-0497-4910-81be-a7876a3f9e03\") " pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:06:27.072166 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:27.072054    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa15671a-0497-4910-81be-a7876a3f9e03-openshift-service-ca-bundle\") pod \"sequence-graph-c559a-57c5445779-r4zfm\" (UID: \"aa15671a-0497-4910-81be-a7876a3f9e03\") " pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:06:27.072166 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:27.072107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa15671a-0497-4910-81be-a7876a3f9e03-proxy-tls\") pod \"sequence-graph-c559a-57c5445779-r4zfm\" (UID: \"aa15671a-0497-4910-81be-a7876a3f9e03\") " pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:06:27.072767 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:27.072741    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa15671a-0497-4910-81be-a7876a3f9e03-openshift-service-ca-bundle\") pod \"sequence-graph-c559a-57c5445779-r4zfm\" (UID: \"aa15671a-0497-4910-81be-a7876a3f9e03\") " pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:06:27.074470 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:27.074449    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa15671a-0497-4910-81be-a7876a3f9e03-proxy-tls\") pod \"sequence-graph-c559a-57c5445779-r4zfm\" (UID: \"aa15671a-0497-4910-81be-a7876a3f9e03\") " pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:06:27.116540 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:27.116517    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:06:27.258333 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:27.258302    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"]
Jul 02 14:06:27.260601 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:06:27.260572    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa15671a_0497_4910_81be_a7876a3f9e03.slice/crio-a52371dd50f7b56d3bae62ae03202b9ba7189d5acc04fda4174821690048eb5a WatchSource:0}: Error finding container a52371dd50f7b56d3bae62ae03202b9ba7189d5acc04fda4174821690048eb5a: Status 404 returned error can't find the container with id a52371dd50f7b56d3bae62ae03202b9ba7189d5acc04fda4174821690048eb5a
Jul 02 14:06:28.182054 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:28.182020    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" event={"ID":"aa15671a-0497-4910-81be-a7876a3f9e03","Type":"ContainerStarted","Data":"36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa"}
Jul 02 14:06:28.182454 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:28.182070    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" event={"ID":"aa15671a-0497-4910-81be-a7876a3f9e03","Type":"ContainerStarted","Data":"a52371dd50f7b56d3bae62ae03202b9ba7189d5acc04fda4174821690048eb5a"}
Jul 02 14:06:28.182454 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:28.182122    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:06:28.204535 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:28.199909    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" podStartSLOduration=2.199890791 podStartE2EDuration="2.199890791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:06:28.198731337 +0000 UTC m=+1325.394354845" watchObservedRunningTime="2026-07-02 14:06:28.199890791 +0000 UTC m=+1325.395514302"
Jul 02 14:06:28.977319 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:28.977285    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:06:34.192654 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:34.192626    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:06:36.884801 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:36.884765    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"]
Jul 02 14:06:36.885303 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:36.885023    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" podUID="aa15671a-0497-4910-81be-a7876a3f9e03" containerName="sequence-graph-c559a" containerID="cri-o://36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa" gracePeriod=30
Jul 02 14:06:37.011864 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.011833    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"]
Jul 02 14:06:37.012194 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.012123    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kube-rbac-proxy" containerID="cri-o://819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd" gracePeriod=30
Jul 02 14:06:37.012775 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.012727    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kserve-container" containerID="cri-o://87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d" gracePeriod=30
Jul 02 14:06:37.018110 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.018085    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"]
Jul 02 14:06:37.022081 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.022059    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:37.024482 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.024461    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6d218-kube-rbac-proxy-sar-config\""
Jul 02 14:06:37.024584 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.024461    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6d218-predictor-serving-cert\""
Jul 02 14:06:37.031285 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.031257    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"]
Jul 02 14:06:37.157042 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.156975    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") " pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:37.157042 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.157014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fa21fbc-9387-4a9b-a614-cea780b794e3-proxy-tls\") pod \"success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") " pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:37.157250 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.157077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bcj\" (UniqueName: \"kubernetes.io/projected/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-api-access-k7bcj\") pod \"success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") " pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:37.224084 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.224052    2574 generic.go:356] "Generic (PLEG): container finished" podID="915650ad-3f44-487c-802d-23748fce727c" containerID="819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd" exitCode=2
Jul 02 14:06:37.224311 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.224119    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" event={"ID":"915650ad-3f44-487c-802d-23748fce727c","Type":"ContainerDied","Data":"819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd"}
Jul 02 14:06:37.258377 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.258342    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bcj\" (UniqueName: \"kubernetes.io/projected/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-api-access-k7bcj\") pod \"success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") " pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:37.258519 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.258486    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") " pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:37.258574 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.258521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fa21fbc-9387-4a9b-a614-cea780b794e3-proxy-tls\") pod \"success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") " pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:37.259237 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.259214    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") " pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:37.260955 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.260936    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fa21fbc-9387-4a9b-a614-cea780b794e3-proxy-tls\") pod \"success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") " pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:37.267221 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.267197    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bcj\" (UniqueName: \"kubernetes.io/projected/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-api-access-k7bcj\") pod \"success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") " pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:37.337452 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.337427    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:37.467517 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:37.467444    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"]
Jul 02 14:06:37.469776 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:06:37.469746    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa21fbc_9387_4a9b_a614_cea780b794e3.slice/crio-7c917150d11b94dd82237ef8fe6e190c1890376e68954f8557365bd902f3f667 WatchSource:0}: Error finding container 7c917150d11b94dd82237ef8fe6e190c1890376e68954f8557365bd902f3f667: Status 404 returned error can't find the container with id 7c917150d11b94dd82237ef8fe6e190c1890376e68954f8557365bd902f3f667
Jul 02 14:06:38.231059 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:38.231011    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" event={"ID":"5fa21fbc-9387-4a9b-a614-cea780b794e3","Type":"ContainerStarted","Data":"ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47"}
Jul 02 14:06:38.231566 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:38.231092    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" event={"ID":"5fa21fbc-9387-4a9b-a614-cea780b794e3","Type":"ContainerStarted","Data":"51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026"}
Jul 02 14:06:38.231566 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:38.231126    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" event={"ID":"5fa21fbc-9387-4a9b-a614-cea780b794e3","Type":"ContainerStarted","Data":"7c917150d11b94dd82237ef8fe6e190c1890376e68954f8557365bd902f3f667"}
Jul 02 14:06:38.231566 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:38.231282    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:38.250828 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:38.250783    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" podStartSLOduration=2.2507691960000002 podStartE2EDuration="2.250769196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:06:38.248345162 +0000 UTC m=+1335.443968669" watchObservedRunningTime="2026-07-02 14:06:38.250769196 +0000 UTC m=+1335.446392702"
Jul 02 14:06:39.190921 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:39.190887    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" podUID="aa15671a-0497-4910-81be-a7876a3f9e03" containerName="sequence-graph-c559a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:06:39.236943 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:39.236913    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:39.238426 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:39.238396    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 02 14:06:40.161929 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.161903    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:06:40.181512 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.181451    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/915650ad-3f44-487c-802d-23748fce727c-kube-rbac-proxy-sar-config\") pod \"915650ad-3f44-487c-802d-23748fce727c\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") "
Jul 02 14:06:40.181651 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.181530    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nk5h\" (UniqueName: \"kubernetes.io/projected/915650ad-3f44-487c-802d-23748fce727c-kube-api-access-8nk5h\") pod \"915650ad-3f44-487c-802d-23748fce727c\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") "
Jul 02 14:06:40.181709 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.181651    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915650ad-3f44-487c-802d-23748fce727c-proxy-tls\") pod \"915650ad-3f44-487c-802d-23748fce727c\" (UID: \"915650ad-3f44-487c-802d-23748fce727c\") "
Jul 02 14:06:40.181933 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.181906    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915650ad-3f44-487c-802d-23748fce727c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "915650ad-3f44-487c-802d-23748fce727c" (UID: "915650ad-3f44-487c-802d-23748fce727c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:06:40.183795 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.183771    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/915650ad-3f44-487c-802d-23748fce727c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "915650ad-3f44-487c-802d-23748fce727c" (UID: "915650ad-3f44-487c-802d-23748fce727c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:06:40.183875 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.183801    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915650ad-3f44-487c-802d-23748fce727c-kube-api-access-8nk5h" (OuterVolumeSpecName: "kube-api-access-8nk5h") pod "915650ad-3f44-487c-802d-23748fce727c" (UID: "915650ad-3f44-487c-802d-23748fce727c"). InnerVolumeSpecName "kube-api-access-8nk5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:06:40.242774 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.242740    2574 generic.go:356] "Generic (PLEG): container finished" podID="915650ad-3f44-487c-802d-23748fce727c" containerID="87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d" exitCode=0
Jul 02 14:06:40.243236 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.242770    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" event={"ID":"915650ad-3f44-487c-802d-23748fce727c","Type":"ContainerDied","Data":"87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d"}
Jul 02 14:06:40.243236 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.242797    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"
Jul 02 14:06:40.243236 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.242847    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667" event={"ID":"915650ad-3f44-487c-802d-23748fce727c","Type":"ContainerDied","Data":"8a6cf040496fe2f30574fad133da26e835d55c71ef2b69e7dafed28c0b5ae225"}
Jul 02 14:06:40.243236 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.242884    2574 scope.go:117] "RemoveContainer" containerID="819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd"
Jul 02 14:06:40.243651 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.243617    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 02 14:06:40.254002 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.253920    2574 scope.go:117] "RemoveContainer" containerID="87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d"
Jul 02 14:06:40.261745 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.261724    2574 scope.go:117] "RemoveContainer" containerID="819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd"
Jul 02 14:06:40.261999 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:06:40.261983    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd\": container with ID starting with 819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd not found: ID does not exist" containerID="819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd"
Jul 02 14:06:40.262045 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.262007    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd"} err="failed to get container status \"819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd\": rpc error: code = NotFound desc = could not find container \"819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd\": container with ID starting with 819fe0aa67fdb27fa640e33e9674e18d07f3969ec8527ffc15710582cff227bd not found: ID does not exist"
Jul 02 14:06:40.262045 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.262024    2574 scope.go:117] "RemoveContainer" containerID="87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d"
Jul 02 14:06:40.262284 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:06:40.262267    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d\": container with ID starting with 87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d not found: ID does not exist" containerID="87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d"
Jul 02 14:06:40.262333 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.262290    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d"} err="failed to get container status \"87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d\": rpc error: code = NotFound desc = could not find container \"87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d\": container with ID starting with 87f36e19f10b56660bda7731277136231433e14aaa61ca4bebc52762e658229d not found: ID does not exist"
Jul 02 14:06:40.271267 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.271246    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"]
Jul 02 14:06:40.274632 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.274616    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c559a-predictor-665765c694-bj667"]
Jul 02 14:06:40.282299 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.282273    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/915650ad-3f44-487c-802d-23748fce727c-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:06:40.282299 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.282293    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/915650ad-3f44-487c-802d-23748fce727c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:06:40.282464 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.282304    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nk5h\" (UniqueName: \"kubernetes.io/projected/915650ad-3f44-487c-802d-23748fce727c-kube-api-access-8nk5h\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:06:40.911537 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.911506    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"]
Jul 02 14:06:40.911839 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.911827    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kserve-container"
Jul 02 14:06:40.911839 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.911841    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kserve-container"
Jul 02 14:06:40.911934 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.911864    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kube-rbac-proxy"
Jul 02 14:06:40.911934 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.911869    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kube-rbac-proxy"
Jul 02 14:06:40.911934 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.911920    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kserve-container"
Jul 02 14:06:40.911934 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.911930    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="915650ad-3f44-487c-802d-23748fce727c" containerName="kube-rbac-proxy"
Jul 02 14:06:40.916258 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.916232    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:06:40.918539 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.918513    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-7b7ab-serving-cert\""
Jul 02 14:06:40.918660 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.918539    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-7b7ab-kube-rbac-proxy-sar-config\""
Jul 02 14:06:40.924949 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.924927    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"]
Jul 02 14:06:40.986572 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.986549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e41de04-13ec-487c-8f26-eb5601f5b713-openshift-service-ca-bundle\") pod \"ensemble-graph-7b7ab-9fcfc967-k4fg6\" (UID: \"6e41de04-13ec-487c-8f26-eb5601f5b713\") " pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:06:40.986706 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:40.986592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e41de04-13ec-487c-8f26-eb5601f5b713-proxy-tls\") pod \"ensemble-graph-7b7ab-9fcfc967-k4fg6\" (UID: \"6e41de04-13ec-487c-8f26-eb5601f5b713\") " pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:06:41.087785 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:41.087749    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e41de04-13ec-487c-8f26-eb5601f5b713-openshift-service-ca-bundle\") pod \"ensemble-graph-7b7ab-9fcfc967-k4fg6\" (UID: \"6e41de04-13ec-487c-8f26-eb5601f5b713\") " pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:06:41.087979 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:41.087841    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e41de04-13ec-487c-8f26-eb5601f5b713-proxy-tls\") pod \"ensemble-graph-7b7ab-9fcfc967-k4fg6\" (UID: \"6e41de04-13ec-487c-8f26-eb5601f5b713\") " pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:06:41.088052 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:06:41.087987    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-7b7ab-serving-cert: secret "ensemble-graph-7b7ab-serving-cert" not found
Jul 02 14:06:41.088128 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:06:41.088053    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e41de04-13ec-487c-8f26-eb5601f5b713-proxy-tls podName:6e41de04-13ec-487c-8f26-eb5601f5b713 nodeName:}" failed. No retries permitted until 2026-07-02 14:06:41.588036754 +0000 UTC m=+1338.783660239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6e41de04-13ec-487c-8f26-eb5601f5b713-proxy-tls") pod "ensemble-graph-7b7ab-9fcfc967-k4fg6" (UID: "6e41de04-13ec-487c-8f26-eb5601f5b713") : secret "ensemble-graph-7b7ab-serving-cert" not found
Jul 02 14:06:41.088472 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:41.088448    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e41de04-13ec-487c-8f26-eb5601f5b713-openshift-service-ca-bundle\") pod \"ensemble-graph-7b7ab-9fcfc967-k4fg6\" (UID: \"6e41de04-13ec-487c-8f26-eb5601f5b713\") " pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:06:41.351111 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:41.351032    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="915650ad-3f44-487c-802d-23748fce727c" path="/var/lib/kubelet/pods/915650ad-3f44-487c-802d-23748fce727c/volumes"
Jul 02 14:06:41.592563 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:41.592535    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e41de04-13ec-487c-8f26-eb5601f5b713-proxy-tls\") pod \"ensemble-graph-7b7ab-9fcfc967-k4fg6\" (UID: \"6e41de04-13ec-487c-8f26-eb5601f5b713\") " pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:06:41.594830 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:41.594800    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e41de04-13ec-487c-8f26-eb5601f5b713-proxy-tls\") pod \"ensemble-graph-7b7ab-9fcfc967-k4fg6\" (UID: \"6e41de04-13ec-487c-8f26-eb5601f5b713\") " pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:06:41.827096 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:41.827060    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:06:41.954369 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:41.954346    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"]
Jul 02 14:06:41.956638 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:06:41.956611    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e41de04_13ec_487c_8f26_eb5601f5b713.slice/crio-67525873a3dede5453d8ddf1ae628aba204efa476af1e1df1c108f1374cdbbec WatchSource:0}: Error finding container 67525873a3dede5453d8ddf1ae628aba204efa476af1e1df1c108f1374cdbbec: Status 404 returned error can't find the container with id 67525873a3dede5453d8ddf1ae628aba204efa476af1e1df1c108f1374cdbbec
Jul 02 14:06:42.253898 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:42.253864    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" event={"ID":"6e41de04-13ec-487c-8f26-eb5601f5b713","Type":"ContainerStarted","Data":"ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a"}
Jul 02 14:06:42.254083 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:42.253914    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" event={"ID":"6e41de04-13ec-487c-8f26-eb5601f5b713","Type":"ContainerStarted","Data":"67525873a3dede5453d8ddf1ae628aba204efa476af1e1df1c108f1374cdbbec"}
Jul 02 14:06:42.254083 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:42.253969    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:06:42.274341 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:42.274296    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" podStartSLOduration=2.274283227 podStartE2EDuration="2.274283227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:06:42.272747604 +0000 UTC m=+1339.468371112" watchObservedRunningTime="2026-07-02 14:06:42.274283227 +0000 UTC m=+1339.469906736"
Jul 02 14:06:44.191037 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:44.190999    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" podUID="aa15671a-0497-4910-81be-a7876a3f9e03" containerName="sequence-graph-c559a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:06:45.251156 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:45.251115    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:06:45.251610 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:45.251587    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 02 14:06:48.264733 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:48.264705    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:06:49.191088 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:49.191050    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" podUID="aa15671a-0497-4910-81be-a7876a3f9e03" containerName="sequence-graph-c559a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:06:49.191282 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:49.191187    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:06:54.190795 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:54.190753    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" podUID="aa15671a-0497-4910-81be-a7876a3f9e03" containerName="sequence-graph-c559a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:06:55.252309 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:55.252275    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 02 14:06:59.191122 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:06:59.191086    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" podUID="aa15671a-0497-4910-81be-a7876a3f9e03" containerName="sequence-graph-c559a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:07:04.191359 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:04.191325    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" podUID="aa15671a-0497-4910-81be-a7876a3f9e03" containerName="sequence-graph-c559a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:07:05.252069 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:05.252027    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 02 14:07:07.039083 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.039053    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:07:07.108032 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.107998    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa15671a-0497-4910-81be-a7876a3f9e03-openshift-service-ca-bundle\") pod \"aa15671a-0497-4910-81be-a7876a3f9e03\" (UID: \"aa15671a-0497-4910-81be-a7876a3f9e03\") "
Jul 02 14:07:07.108222 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.108045    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa15671a-0497-4910-81be-a7876a3f9e03-proxy-tls\") pod \"aa15671a-0497-4910-81be-a7876a3f9e03\" (UID: \"aa15671a-0497-4910-81be-a7876a3f9e03\") "
Jul 02 14:07:07.108470 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.108439    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa15671a-0497-4910-81be-a7876a3f9e03-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "aa15671a-0497-4910-81be-a7876a3f9e03" (UID: "aa15671a-0497-4910-81be-a7876a3f9e03"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:07:07.110089 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.110057    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa15671a-0497-4910-81be-a7876a3f9e03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa15671a-0497-4910-81be-a7876a3f9e03" (UID: "aa15671a-0497-4910-81be-a7876a3f9e03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:07:07.209698 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.209668    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa15671a-0497-4910-81be-a7876a3f9e03-openshift-service-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:07:07.209698 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.209693    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa15671a-0497-4910-81be-a7876a3f9e03-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:07:07.362872 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.362842    2574 generic.go:356] "Generic (PLEG): container finished" podID="aa15671a-0497-4910-81be-a7876a3f9e03" containerID="36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa" exitCode=0
Jul 02 14:07:07.363016 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.362899    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"
Jul 02 14:07:07.363016 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.362917    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" event={"ID":"aa15671a-0497-4910-81be-a7876a3f9e03","Type":"ContainerDied","Data":"36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa"}
Jul 02 14:07:07.363016 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.362966    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm" event={"ID":"aa15671a-0497-4910-81be-a7876a3f9e03","Type":"ContainerDied","Data":"a52371dd50f7b56d3bae62ae03202b9ba7189d5acc04fda4174821690048eb5a"}
Jul 02 14:07:07.363016 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.362989    2574 scope.go:117] "RemoveContainer" containerID="36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa"
Jul 02 14:07:07.371493 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.371475    2574 scope.go:117] "RemoveContainer" containerID="36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa"
Jul 02 14:07:07.371715 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:07:07.371700    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa\": container with ID starting with 36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa not found: ID does not exist" containerID="36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa"
Jul 02 14:07:07.371764 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.371722    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa"} err="failed to get container status \"36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa\": rpc error: code = NotFound desc = could not find container \"36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa\": container with ID starting with 36c2ca87ac1c8fb04212349a036d9d548507f0b0fd0657901a4194f462fddffa not found: ID does not exist"
Jul 02 14:07:07.381104 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.381081    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"]
Jul 02 14:07:07.386693 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:07.386674    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-c559a-57c5445779-r4zfm"]
Jul 02 14:07:09.350593 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:09.350562    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa15671a-0497-4910-81be-a7876a3f9e03" path="/var/lib/kubelet/pods/aa15671a-0497-4910-81be-a7876a3f9e03/volumes"
Jul 02 14:07:15.251908 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:15.251865    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 02 14:07:25.252035 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:25.252007    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:07:37.108877 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.108839    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"]
Jul 02 14:07:37.109347 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.109332    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa15671a-0497-4910-81be-a7876a3f9e03" containerName="sequence-graph-c559a"
Jul 02 14:07:37.109394 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.109349    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa15671a-0497-4910-81be-a7876a3f9e03" containerName="sequence-graph-c559a"
Jul 02 14:07:37.109429 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.109411    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa15671a-0497-4910-81be-a7876a3f9e03" containerName="sequence-graph-c559a"
Jul 02 14:07:37.113396 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.113382    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:07:37.116014 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.115989    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-6d218-kube-rbac-proxy-sar-config\""
Jul 02 14:07:37.116185 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.116033    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-6d218-serving-cert\""
Jul 02 14:07:37.122870 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.122843    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"]
Jul 02 14:07:37.166915 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.166889    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9d8706-f755-42c1-9fb7-8c31e211345d-proxy-tls\") pod \"sequence-graph-6d218-57587c6648-hpj45\" (UID: \"8d9d8706-f755-42c1-9fb7-8c31e211345d\") " pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:07:37.167055 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.166933    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9d8706-f755-42c1-9fb7-8c31e211345d-openshift-service-ca-bundle\") pod \"sequence-graph-6d218-57587c6648-hpj45\" (UID: \"8d9d8706-f755-42c1-9fb7-8c31e211345d\") " pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:07:37.267804 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.267769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9d8706-f755-42c1-9fb7-8c31e211345d-proxy-tls\") pod \"sequence-graph-6d218-57587c6648-hpj45\" (UID: \"8d9d8706-f755-42c1-9fb7-8c31e211345d\") " pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:07:37.267804 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.267812    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9d8706-f755-42c1-9fb7-8c31e211345d-openshift-service-ca-bundle\") pod \"sequence-graph-6d218-57587c6648-hpj45\" (UID: \"8d9d8706-f755-42c1-9fb7-8c31e211345d\") " pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:07:37.268459 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.268431    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9d8706-f755-42c1-9fb7-8c31e211345d-openshift-service-ca-bundle\") pod \"sequence-graph-6d218-57587c6648-hpj45\" (UID: \"8d9d8706-f755-42c1-9fb7-8c31e211345d\") " pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:07:37.270148 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.270114    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9d8706-f755-42c1-9fb7-8c31e211345d-proxy-tls\") pod \"sequence-graph-6d218-57587c6648-hpj45\" (UID: \"8d9d8706-f755-42c1-9fb7-8c31e211345d\") " pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:07:37.423550 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.423526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:07:37.555334 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:37.555316    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"]
Jul 02 14:07:37.557568 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:07:37.557542    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9d8706_f755_42c1_9fb7_8c31e211345d.slice/crio-ce181cc3a24d9661183c36ef23216353c939721f38806bd9651739f39da0c19e WatchSource:0}: Error finding container ce181cc3a24d9661183c36ef23216353c939721f38806bd9651739f39da0c19e: Status 404 returned error can't find the container with id ce181cc3a24d9661183c36ef23216353c939721f38806bd9651739f39da0c19e
Jul 02 14:07:38.497107 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:38.497073    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" event={"ID":"8d9d8706-f755-42c1-9fb7-8c31e211345d","Type":"ContainerStarted","Data":"659c025279176840838bd5839e125a8fb9fa4a9c46e084748135ec64ddc5c011"}
Jul 02 14:07:38.497523 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:38.497129    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" event={"ID":"8d9d8706-f755-42c1-9fb7-8c31e211345d","Type":"ContainerStarted","Data":"ce181cc3a24d9661183c36ef23216353c939721f38806bd9651739f39da0c19e"}
Jul 02 14:07:38.497523 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:38.497179    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:07:38.514432 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:38.514382    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" podStartSLOduration=1.5143662839999998 podStartE2EDuration="1.514366284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:07:38.513518987 +0000 UTC m=+1395.709142495" watchObservedRunningTime="2026-07-02 14:07:38.514366284 +0000 UTC m=+1395.709989791"
Jul 02 14:07:44.508128 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:07:44.508098    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:09:23.387240 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:09:23.387216    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:09:23.391120 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:09:23.391095    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:09:23.393677 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:09:23.393658    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:09:23.397517 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:09:23.397503    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:14:23.414227 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:23.414201    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:14:23.418921 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:23.418899    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:14:23.420606 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:23.420585    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:14:23.425272 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:23.425254    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:14:55.676222 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:55.676189    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"]
Jul 02 14:14:55.676708 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:55.676387    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" podUID="6e41de04-13ec-487c-8f26-eb5601f5b713" containerName="ensemble-graph-7b7ab" containerID="cri-o://ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a" gracePeriod=30
Jul 02 14:14:55.877637 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:55.877597    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"]
Jul 02 14:14:55.881078 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:55.881056    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:55.883625 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:55.883606    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-09121-kube-rbac-proxy-sar-config\""
Jul 02 14:14:55.883745 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:55.883683    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-09121-predictor-serving-cert\""
Jul 02 14:14:55.893589 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:55.893569    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"]
Jul 02 14:14:55.898410 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:55.898385    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"]
Jul 02 14:14:55.898636 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:55.898612    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kserve-container" containerID="cri-o://c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f" gracePeriod=30
Jul 02 14:14:55.898742 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:55.898712    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kube-rbac-proxy" containerID="cri-o://dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4" gracePeriod=30
Jul 02 14:14:56.010720 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.010697    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7z4k\" (UniqueName: \"kubernetes.io/projected/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-api-access-q7z4k\") pod \"success-200-isvc-09121-predictor-66bcd6b46-k2xnm\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") " pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:56.010793 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.010750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-09121-predictor-66bcd6b46-k2xnm\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") " pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:56.010847 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.010829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae79875-c7f8-4433-ba70-8d2d76b10627-proxy-tls\") pod \"success-200-isvc-09121-predictor-66bcd6b46-k2xnm\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") " pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:56.111450 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.111409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-09121-predictor-66bcd6b46-k2xnm\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") " pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:56.111626 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.111573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae79875-c7f8-4433-ba70-8d2d76b10627-proxy-tls\") pod \"success-200-isvc-09121-predictor-66bcd6b46-k2xnm\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") " pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:56.111685 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.111660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7z4k\" (UniqueName: \"kubernetes.io/projected/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-api-access-q7z4k\") pod \"success-200-isvc-09121-predictor-66bcd6b46-k2xnm\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") " pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:56.112252 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.112227    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-09121-predictor-66bcd6b46-k2xnm\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") " pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:56.114037 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.114013    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae79875-c7f8-4433-ba70-8d2d76b10627-proxy-tls\") pod \"success-200-isvc-09121-predictor-66bcd6b46-k2xnm\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") " pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:56.121879 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.121851    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7z4k\" (UniqueName: \"kubernetes.io/projected/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-api-access-q7z4k\") pod \"success-200-isvc-09121-predictor-66bcd6b46-k2xnm\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") " pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:56.191770 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.191739    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:56.325763 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.325732    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"]
Jul 02 14:14:56.328860 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:14:56.328832    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae79875_c7f8_4433_ba70_8d2d76b10627.slice/crio-927bca7a518c63df1bb9b19fd30e4d7c4be1565d012659626c46195461b80a5d WatchSource:0}: Error finding container 927bca7a518c63df1bb9b19fd30e4d7c4be1565d012659626c46195461b80a5d: Status 404 returned error can't find the container with id 927bca7a518c63df1bb9b19fd30e4d7c4be1565d012659626c46195461b80a5d
Jul 02 14:14:56.404826 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.404803    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" event={"ID":"4ae79875-c7f8-4433-ba70-8d2d76b10627","Type":"ContainerStarted","Data":"5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768"}
Jul 02 14:14:56.404946 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.404864    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" event={"ID":"4ae79875-c7f8-4433-ba70-8d2d76b10627","Type":"ContainerStarted","Data":"927bca7a518c63df1bb9b19fd30e4d7c4be1565d012659626c46195461b80a5d"}
Jul 02 14:14:56.406577 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.406554    2574 generic.go:356] "Generic (PLEG): container finished" podID="285570f8-0db8-42d5-8404-1e07570d46a0" containerID="dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4" exitCode=2
Jul 02 14:14:56.406682 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:56.406602    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" event={"ID":"285570f8-0db8-42d5-8404-1e07570d46a0","Type":"ContainerDied","Data":"dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4"}
Jul 02 14:14:57.412970 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:57.412917    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" event={"ID":"4ae79875-c7f8-4433-ba70-8d2d76b10627","Type":"ContainerStarted","Data":"76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371"}
Jul 02 14:14:57.413445 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:57.413159    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:57.434265 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:57.434217    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" podStartSLOduration=2.434202954 podStartE2EDuration="2.434202954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:14:57.431829661 +0000 UTC m=+1834.627453175" watchObservedRunningTime="2026-07-02 14:14:57.434202954 +0000 UTC m=+1834.629826460"
Jul 02 14:14:58.263008 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:58.262970    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" podUID="6e41de04-13ec-487c-8f26-eb5601f5b713" containerName="ensemble-graph-7b7ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:14:58.417987 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:58.417953    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:14:58.419483 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:58.419455    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 02 14:14:58.941755 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:58.941727    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:14:59.035295 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.035214    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285570f8-0db8-42d5-8404-1e07570d46a0-proxy-tls\") pod \"285570f8-0db8-42d5-8404-1e07570d46a0\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") "
Jul 02 14:14:59.035295 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.035252    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7lpz\" (UniqueName: \"kubernetes.io/projected/285570f8-0db8-42d5-8404-1e07570d46a0-kube-api-access-l7lpz\") pod \"285570f8-0db8-42d5-8404-1e07570d46a0\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") "
Jul 02 14:14:59.035523 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.035315    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285570f8-0db8-42d5-8404-1e07570d46a0-kube-rbac-proxy-sar-config\") pod \"285570f8-0db8-42d5-8404-1e07570d46a0\" (UID: \"285570f8-0db8-42d5-8404-1e07570d46a0\") "
Jul 02 14:14:59.035744 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.035721    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285570f8-0db8-42d5-8404-1e07570d46a0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "285570f8-0db8-42d5-8404-1e07570d46a0" (UID: "285570f8-0db8-42d5-8404-1e07570d46a0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:14:59.037394 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.037370    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285570f8-0db8-42d5-8404-1e07570d46a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "285570f8-0db8-42d5-8404-1e07570d46a0" (UID: "285570f8-0db8-42d5-8404-1e07570d46a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:14:59.037459 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.037423    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285570f8-0db8-42d5-8404-1e07570d46a0-kube-api-access-l7lpz" (OuterVolumeSpecName: "kube-api-access-l7lpz") pod "285570f8-0db8-42d5-8404-1e07570d46a0" (UID: "285570f8-0db8-42d5-8404-1e07570d46a0"). InnerVolumeSpecName "kube-api-access-l7lpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:14:59.136022 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.135984    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285570f8-0db8-42d5-8404-1e07570d46a0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:14:59.136022 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.136017    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285570f8-0db8-42d5-8404-1e07570d46a0-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:14:59.136022 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.136031    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7lpz\" (UniqueName: \"kubernetes.io/projected/285570f8-0db8-42d5-8404-1e07570d46a0-kube-api-access-l7lpz\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:14:59.423520 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.423490    2574 generic.go:356] "Generic (PLEG): container finished" podID="285570f8-0db8-42d5-8404-1e07570d46a0" containerID="c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f" exitCode=0
Jul 02 14:14:59.423926 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.423538    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"
Jul 02 14:14:59.423926 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.423586    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" event={"ID":"285570f8-0db8-42d5-8404-1e07570d46a0","Type":"ContainerDied","Data":"c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f"}
Jul 02 14:14:59.423926 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.423647    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx" event={"ID":"285570f8-0db8-42d5-8404-1e07570d46a0","Type":"ContainerDied","Data":"f668a3af5eaeb9f69be8beef188868c3a4bc942ef93b71d7c4c1f82f9b8edb02"}
Jul 02 14:14:59.423926 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.423671    2574 scope.go:117] "RemoveContainer" containerID="dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4"
Jul 02 14:14:59.424185 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.424162    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 02 14:14:59.433505 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.433489    2574 scope.go:117] "RemoveContainer" containerID="c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f"
Jul 02 14:14:59.440753 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.440737    2574 scope.go:117] "RemoveContainer" containerID="dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4"
Jul 02 14:14:59.440981 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:14:59.440963    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4\": container with ID starting with dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4 not found: ID does not exist" containerID="dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4"
Jul 02 14:14:59.441041 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.440988    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4"} err="failed to get container status \"dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4\": rpc error: code = NotFound desc = could not find container \"dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4\": container with ID starting with dcb86585021b10fb2af00c81a6270e5f1e2ad7490cdaaf3a52a59c7305a05ef4 not found: ID does not exist"
Jul 02 14:14:59.441041 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.441004    2574 scope.go:117] "RemoveContainer" containerID="c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f"
Jul 02 14:14:59.441282 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:14:59.441264    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f\": container with ID starting with c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f not found: ID does not exist" containerID="c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f"
Jul 02 14:14:59.441367 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.441290    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f"} err="failed to get container status \"c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f\": rpc error: code = NotFound desc = could not find container \"c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f\": container with ID starting with c179d76145213510e166dfaee02ee04f83be9f563c27d4122867bcc33173b44f not found: ID does not exist"
Jul 02 14:14:59.444460 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.444438    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"]
Jul 02 14:14:59.448985 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:14:59.448967    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b7ab-predictor-6cbf754778-jfbcx"]
Jul 02 14:15:01.356245 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:01.356208    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" path="/var/lib/kubelet/pods/285570f8-0db8-42d5-8404-1e07570d46a0/volumes"
Jul 02 14:15:03.262764 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:03.262729    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" podUID="6e41de04-13ec-487c-8f26-eb5601f5b713" containerName="ensemble-graph-7b7ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:15:04.429192 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:04.429158    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:15:04.429684 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:04.429659    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 02 14:15:08.263114 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:08.263076    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" podUID="6e41de04-13ec-487c-8f26-eb5601f5b713" containerName="ensemble-graph-7b7ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:15:08.263571 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:08.263220    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:15:13.263030 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:13.262991    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" podUID="6e41de04-13ec-487c-8f26-eb5601f5b713" containerName="ensemble-graph-7b7ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:15:14.429870 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:14.429827    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 02 14:15:18.263160 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:18.263104    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" podUID="6e41de04-13ec-487c-8f26-eb5601f5b713" containerName="ensemble-graph-7b7ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:15:23.262881 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:23.262837    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" podUID="6e41de04-13ec-487c-8f26-eb5601f5b713" containerName="ensemble-graph-7b7ab" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:15:24.429694 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:24.429653    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 02 14:15:25.832849 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:25.832825    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:15:25.857847 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:25.857817    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e41de04-13ec-487c-8f26-eb5601f5b713-openshift-service-ca-bundle\") pod \"6e41de04-13ec-487c-8f26-eb5601f5b713\" (UID: \"6e41de04-13ec-487c-8f26-eb5601f5b713\") "
Jul 02 14:15:25.857970 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:25.857862    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e41de04-13ec-487c-8f26-eb5601f5b713-proxy-tls\") pod \"6e41de04-13ec-487c-8f26-eb5601f5b713\" (UID: \"6e41de04-13ec-487c-8f26-eb5601f5b713\") "
Jul 02 14:15:25.858156 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:25.858122    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e41de04-13ec-487c-8f26-eb5601f5b713-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6e41de04-13ec-487c-8f26-eb5601f5b713" (UID: "6e41de04-13ec-487c-8f26-eb5601f5b713"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:15:25.859771 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:25.859746    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e41de04-13ec-487c-8f26-eb5601f5b713-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e41de04-13ec-487c-8f26-eb5601f5b713" (UID: "6e41de04-13ec-487c-8f26-eb5601f5b713"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:15:25.959428 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:25.959368    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e41de04-13ec-487c-8f26-eb5601f5b713-openshift-service-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:15:25.959428 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:25.959391    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e41de04-13ec-487c-8f26-eb5601f5b713-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:15:26.546680 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:26.546648    2574 generic.go:356] "Generic (PLEG): container finished" podID="6e41de04-13ec-487c-8f26-eb5601f5b713" containerID="ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a" exitCode=0
Jul 02 14:15:26.546869 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:26.546688    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"
Jul 02 14:15:26.546869 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:26.546727    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" event={"ID":"6e41de04-13ec-487c-8f26-eb5601f5b713","Type":"ContainerDied","Data":"ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a"}
Jul 02 14:15:26.546869 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:26.546779    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6" event={"ID":"6e41de04-13ec-487c-8f26-eb5601f5b713","Type":"ContainerDied","Data":"67525873a3dede5453d8ddf1ae628aba204efa476af1e1df1c108f1374cdbbec"}
Jul 02 14:15:26.546869 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:26.546809    2574 scope.go:117] "RemoveContainer" containerID="ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a"
Jul 02 14:15:26.555367 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:26.555349    2574 scope.go:117] "RemoveContainer" containerID="ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a"
Jul 02 14:15:26.555597 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:15:26.555581    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a\": container with ID starting with ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a not found: ID does not exist" containerID="ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a"
Jul 02 14:15:26.555640 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:26.555606    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a"} err="failed to get container status \"ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a\": rpc error: code = NotFound desc = could not find container \"ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a\": container with ID starting with ec8c90ba5d52db1a540955fe818219f0089f6b2b02666f2082898aabe106036a not found: ID does not exist"
Jul 02 14:15:26.572738 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:26.572720    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"]
Jul 02 14:15:26.576485 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:26.576467    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-7b7ab-9fcfc967-k4fg6"]
Jul 02 14:15:27.350593 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:27.350561    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e41de04-13ec-487c-8f26-eb5601f5b713" path="/var/lib/kubelet/pods/6e41de04-13ec-487c-8f26-eb5601f5b713/volumes"
Jul 02 14:15:34.429683 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:34.429644    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 02 14:15:44.430856 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:44.430779    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:15:51.677037 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.677004    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"]
Jul 02 14:15:51.677455 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.677255    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" podUID="8d9d8706-f755-42c1-9fb7-8c31e211345d" containerName="sequence-graph-6d218" containerID="cri-o://659c025279176840838bd5839e125a8fb9fa4a9c46e084748135ec64ddc5c011" gracePeriod=30
Jul 02 14:15:51.825251 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.825220    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"]
Jul 02 14:15:51.825468 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.825443    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kserve-container" containerID="cri-o://51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026" gracePeriod=30
Jul 02 14:15:51.825554 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.825483    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kube-rbac-proxy" containerID="cri-o://ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47" gracePeriod=30
Jul 02 14:15:51.856339 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.856310    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"]
Jul 02 14:15:51.856636 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.856625    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kserve-container"
Jul 02 14:15:51.856683 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.856638    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kserve-container"
Jul 02 14:15:51.856683 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.856661    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e41de04-13ec-487c-8f26-eb5601f5b713" containerName="ensemble-graph-7b7ab"
Jul 02 14:15:51.856683 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.856667    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e41de04-13ec-487c-8f26-eb5601f5b713" containerName="ensemble-graph-7b7ab"
Jul 02 14:15:51.856683 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.856675    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kube-rbac-proxy"
Jul 02 14:15:51.856683 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.856681    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kube-rbac-proxy"
Jul 02 14:15:51.856834 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.856738    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kserve-container"
Jul 02 14:15:51.856834 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.856745    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="285570f8-0db8-42d5-8404-1e07570d46a0" containerName="kube-rbac-proxy"
Jul 02 14:15:51.856834 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.856752    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e41de04-13ec-487c-8f26-eb5601f5b713" containerName="ensemble-graph-7b7ab"
Jul 02 14:15:51.861029 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.861010    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:51.864334 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.864308    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8d2ee-predictor-serving-cert\""
Jul 02 14:15:51.864455 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.864436    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8d2ee-kube-rbac-proxy-sar-config\""
Jul 02 14:15:51.871546 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.871524    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"]
Jul 02 14:15:51.982013 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.981980    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkb6w\" (UniqueName: \"kubernetes.io/projected/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-api-access-vkb6w\") pod \"success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") " pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:51.982189 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.982031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f294d0b1-01b6-40de-aef0-c6cdb40552a3-proxy-tls\") pod \"success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") " pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:51.982189 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:51.982115    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") " pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:52.083484 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.083453    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") " pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:52.083663 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.083511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkb6w\" (UniqueName: \"kubernetes.io/projected/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-api-access-vkb6w\") pod \"success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") " pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:52.083663 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.083562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f294d0b1-01b6-40de-aef0-c6cdb40552a3-proxy-tls\") pod \"success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") " pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:52.084116 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.084085    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") " pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:52.085837 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.085815    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f294d0b1-01b6-40de-aef0-c6cdb40552a3-proxy-tls\") pod \"success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") " pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:52.092521 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.092496    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkb6w\" (UniqueName: \"kubernetes.io/projected/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-api-access-vkb6w\") pod \"success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") " pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:52.171260 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.171234    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:52.303662 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.303637    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"]
Jul 02 14:15:52.305628 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:15:52.305601    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf294d0b1_01b6_40de_aef0_c6cdb40552a3.slice/crio-d53b677923a884e15a9c0a093cb8d7db4a7603fa326e9b50b3ff9a59a8be116a WatchSource:0}: Error finding container d53b677923a884e15a9c0a093cb8d7db4a7603fa326e9b50b3ff9a59a8be116a: Status 404 returned error can't find the container with id d53b677923a884e15a9c0a093cb8d7db4a7603fa326e9b50b3ff9a59a8be116a
Jul 02 14:15:52.667221 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.667189    2574 generic.go:356] "Generic (PLEG): container finished" podID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerID="ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47" exitCode=2
Jul 02 14:15:52.667363 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.667264    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" event={"ID":"5fa21fbc-9387-4a9b-a614-cea780b794e3","Type":"ContainerDied","Data":"ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47"}
Jul 02 14:15:52.669040 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.669015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" event={"ID":"f294d0b1-01b6-40de-aef0-c6cdb40552a3","Type":"ContainerStarted","Data":"5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db"}
Jul 02 14:15:52.669183 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.669058    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" event={"ID":"f294d0b1-01b6-40de-aef0-c6cdb40552a3","Type":"ContainerStarted","Data":"ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff"}
Jul 02 14:15:52.669183 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.669079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" event={"ID":"f294d0b1-01b6-40de-aef0-c6cdb40552a3","Type":"ContainerStarted","Data":"d53b677923a884e15a9c0a093cb8d7db4a7603fa326e9b50b3ff9a59a8be116a"}
Jul 02 14:15:52.669183 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.669169    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:52.702080 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:52.702028    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" podStartSLOduration=1.7020125940000002 podStartE2EDuration="1.702012594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:15:52.700991543 +0000 UTC m=+1889.896615050" watchObservedRunningTime="2026-07-02 14:15:52.702012594 +0000 UTC m=+1889.897636100"
Jul 02 14:15:53.673952 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:53.673917    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:53.675168 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:53.675120    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 14:15:54.505414 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:54.505373    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" podUID="8d9d8706-f755-42c1-9fb7-8c31e211345d" containerName="sequence-graph-6d218" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:15:54.679025 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:54.678987    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 14:15:54.879579 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:54.879557    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:15:55.007941 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.007910    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7bcj\" (UniqueName: \"kubernetes.io/projected/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-api-access-k7bcj\") pod \"5fa21fbc-9387-4a9b-a614-cea780b794e3\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") "
Jul 02 14:15:55.008153 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.007960    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-rbac-proxy-sar-config\") pod \"5fa21fbc-9387-4a9b-a614-cea780b794e3\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") "
Jul 02 14:15:55.008153 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.008014    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fa21fbc-9387-4a9b-a614-cea780b794e3-proxy-tls\") pod \"5fa21fbc-9387-4a9b-a614-cea780b794e3\" (UID: \"5fa21fbc-9387-4a9b-a614-cea780b794e3\") "
Jul 02 14:15:55.008425 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.008397    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5fa21fbc-9387-4a9b-a614-cea780b794e3" (UID: "5fa21fbc-9387-4a9b-a614-cea780b794e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:15:55.010003 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.009981    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa21fbc-9387-4a9b-a614-cea780b794e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5fa21fbc-9387-4a9b-a614-cea780b794e3" (UID: "5fa21fbc-9387-4a9b-a614-cea780b794e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:15:55.010154 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.010071    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-api-access-k7bcj" (OuterVolumeSpecName: "kube-api-access-k7bcj") pod "5fa21fbc-9387-4a9b-a614-cea780b794e3" (UID: "5fa21fbc-9387-4a9b-a614-cea780b794e3"). InnerVolumeSpecName "kube-api-access-k7bcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:15:55.109195 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.109089    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7bcj\" (UniqueName: \"kubernetes.io/projected/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-api-access-k7bcj\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:15:55.109195 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.109118    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fa21fbc-9387-4a9b-a614-cea780b794e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:15:55.109195 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.109130    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fa21fbc-9387-4a9b-a614-cea780b794e3-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:15:55.684145 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.684113    2574 generic.go:356] "Generic (PLEG): container finished" podID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerID="51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026" exitCode=0
Jul 02 14:15:55.684562 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.684166    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" event={"ID":"5fa21fbc-9387-4a9b-a614-cea780b794e3","Type":"ContainerDied","Data":"51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026"}
Jul 02 14:15:55.684562 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.684191    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"
Jul 02 14:15:55.684562 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.684226    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl" event={"ID":"5fa21fbc-9387-4a9b-a614-cea780b794e3","Type":"ContainerDied","Data":"7c917150d11b94dd82237ef8fe6e190c1890376e68954f8557365bd902f3f667"}
Jul 02 14:15:55.684562 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.684255    2574 scope.go:117] "RemoveContainer" containerID="ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47"
Jul 02 14:15:55.692379 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.692362    2574 scope.go:117] "RemoveContainer" containerID="51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026"
Jul 02 14:15:55.699468 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.699453    2574 scope.go:117] "RemoveContainer" containerID="ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47"
Jul 02 14:15:55.699720 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:15:55.699701    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47\": container with ID starting with ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47 not found: ID does not exist" containerID="ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47"
Jul 02 14:15:55.699799 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.699734    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47"} err="failed to get container status \"ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47\": rpc error: code = NotFound desc = could not find container \"ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47\": container with ID starting with ed0fee4ec4a37adceea45ac12f3685182b81021c1232e1c320c236946c29ac47 not found: ID does not exist"
Jul 02 14:15:55.699799 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.699759    2574 scope.go:117] "RemoveContainer" containerID="51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026"
Jul 02 14:15:55.700036 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:15:55.700016    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026\": container with ID starting with 51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026 not found: ID does not exist" containerID="51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026"
Jul 02 14:15:55.700075 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.700042    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026"} err="failed to get container status \"51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026\": rpc error: code = NotFound desc = could not find container \"51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026\": container with ID starting with 51c60d9f294d72848b68070e27925d4b991273df27a593cdd515d12871237026 not found: ID does not exist"
Jul 02 14:15:55.714470 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.714448    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"]
Jul 02 14:15:55.730288 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.730270    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6d218-predictor-6fbfd44c8-7cpcl"]
Jul 02 14:15:55.977005 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.976901    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"]
Jul 02 14:15:55.977736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.977708    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kserve-container"
Jul 02 14:15:55.977736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.977738    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kserve-container"
Jul 02 14:15:55.977895 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.977778    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kube-rbac-proxy"
Jul 02 14:15:55.977895 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.977787    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kube-rbac-proxy"
Jul 02 14:15:55.977997 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.977930    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kserve-container"
Jul 02 14:15:55.977997 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.977943    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" containerName="kube-rbac-proxy"
Jul 02 14:15:55.983343 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.983319    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:15:55.987221 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.987010    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-09121-serving-cert\""
Jul 02 14:15:55.987221 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.987039    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-09121-kube-rbac-proxy-sar-config\""
Jul 02 14:15:55.988947 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:55.988914    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"]
Jul 02 14:15:56.116994 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.116960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147221c-2bcd-496d-a48f-e8cd49036e4d-openshift-service-ca-bundle\") pod \"splitter-graph-09121-864f98c7fd-rj8ns\" (UID: \"9147221c-2bcd-496d-a48f-e8cd49036e4d\") " pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:15:56.117210 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.117036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9147221c-2bcd-496d-a48f-e8cd49036e4d-proxy-tls\") pod \"splitter-graph-09121-864f98c7fd-rj8ns\" (UID: \"9147221c-2bcd-496d-a48f-e8cd49036e4d\") " pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:15:56.217691 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.217657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9147221c-2bcd-496d-a48f-e8cd49036e4d-proxy-tls\") pod \"splitter-graph-09121-864f98c7fd-rj8ns\" (UID: \"9147221c-2bcd-496d-a48f-e8cd49036e4d\") " pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:15:56.217872 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.217715    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147221c-2bcd-496d-a48f-e8cd49036e4d-openshift-service-ca-bundle\") pod \"splitter-graph-09121-864f98c7fd-rj8ns\" (UID: \"9147221c-2bcd-496d-a48f-e8cd49036e4d\") " pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:15:56.218384 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.218360    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147221c-2bcd-496d-a48f-e8cd49036e4d-openshift-service-ca-bundle\") pod \"splitter-graph-09121-864f98c7fd-rj8ns\" (UID: \"9147221c-2bcd-496d-a48f-e8cd49036e4d\") " pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:15:56.220021 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.219999    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9147221c-2bcd-496d-a48f-e8cd49036e4d-proxy-tls\") pod \"splitter-graph-09121-864f98c7fd-rj8ns\" (UID: \"9147221c-2bcd-496d-a48f-e8cd49036e4d\") " pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:15:56.293730 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.293668    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:15:56.422083 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.422059    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"]
Jul 02 14:15:56.424403 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:15:56.424377    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9147221c_2bcd_496d_a48f_e8cd49036e4d.slice/crio-5358c4174c10a34727d73e362ab15ad014f31ad91b7b301759d4b190da66ff28 WatchSource:0}: Error finding container 5358c4174c10a34727d73e362ab15ad014f31ad91b7b301759d4b190da66ff28: Status 404 returned error can't find the container with id 5358c4174c10a34727d73e362ab15ad014f31ad91b7b301759d4b190da66ff28
Jul 02 14:15:56.690983 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.690936    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" event={"ID":"9147221c-2bcd-496d-a48f-e8cd49036e4d","Type":"ContainerStarted","Data":"ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063"}
Jul 02 14:15:56.691407 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.691002    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" event={"ID":"9147221c-2bcd-496d-a48f-e8cd49036e4d","Type":"ContainerStarted","Data":"5358c4174c10a34727d73e362ab15ad014f31ad91b7b301759d4b190da66ff28"}
Jul 02 14:15:56.691407 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.691029    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:15:56.711737 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:56.711696    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" podStartSLOduration=1.7116800269999999 podStartE2EDuration="1.711680027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:15:56.710712489 +0000 UTC m=+1893.906335995" watchObservedRunningTime="2026-07-02 14:15:56.711680027 +0000 UTC m=+1893.907303569"
Jul 02 14:15:57.351250 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:57.351218    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fa21fbc-9387-4a9b-a614-cea780b794e3" path="/var/lib/kubelet/pods/5fa21fbc-9387-4a9b-a614-cea780b794e3/volumes"
Jul 02 14:15:59.505937 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:59.505902    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" podUID="8d9d8706-f755-42c1-9fb7-8c31e211345d" containerName="sequence-graph-6d218" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:15:59.685438 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:59.685404    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:15:59.685822 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:15:59.685794    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 14:16:02.707560 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:02.702094    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:16:04.505828 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:04.505789    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" podUID="8d9d8706-f755-42c1-9fb7-8c31e211345d" containerName="sequence-graph-6d218" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:16:04.506231 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:04.505892    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:16:05.994511 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:05.994475    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"]
Jul 02 14:16:05.995064 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:05.994672    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" podUID="9147221c-2bcd-496d-a48f-e8cd49036e4d" containerName="splitter-graph-09121" containerID="cri-o://ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063" gracePeriod=30
Jul 02 14:16:06.133985 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.133948    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"]
Jul 02 14:16:06.134289 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.134261    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kserve-container" containerID="cri-o://5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768" gracePeriod=30
Jul 02 14:16:06.134355 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.134328    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kube-rbac-proxy" containerID="cri-o://76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371" gracePeriod=30
Jul 02 14:16:06.152683 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.152660    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"]
Jul 02 14:16:06.157408 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.157388    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.160304 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.160281    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-78590-predictor-serving-cert\""
Jul 02 14:16:06.160403 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.160366    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-78590-kube-rbac-proxy-sar-config\""
Jul 02 14:16:06.167975 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.167955    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"]
Jul 02 14:16:06.303427 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.303396    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78590-predictor-599b7b7f46-8r44l\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") " pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.303604 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.303446    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vpk\" (UniqueName: \"kubernetes.io/projected/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-api-access-b9vpk\") pod \"success-200-isvc-78590-predictor-599b7b7f46-8r44l\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") " pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.303604 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.303523    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a1816a-ea79-42c5-af42-a3f922f1526e-proxy-tls\") pod \"success-200-isvc-78590-predictor-599b7b7f46-8r44l\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") " pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.404842 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.404807    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a1816a-ea79-42c5-af42-a3f922f1526e-proxy-tls\") pod \"success-200-isvc-78590-predictor-599b7b7f46-8r44l\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") " pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.405009 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.404880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78590-predictor-599b7b7f46-8r44l\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") " pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.405009 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.404908    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vpk\" (UniqueName: \"kubernetes.io/projected/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-api-access-b9vpk\") pod \"success-200-isvc-78590-predictor-599b7b7f46-8r44l\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") " pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.405554 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.405532    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78590-predictor-599b7b7f46-8r44l\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") " pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.407320 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.407296    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a1816a-ea79-42c5-af42-a3f922f1526e-proxy-tls\") pod \"success-200-isvc-78590-predictor-599b7b7f46-8r44l\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") " pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.413988 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.413965    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vpk\" (UniqueName: \"kubernetes.io/projected/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-api-access-b9vpk\") pod \"success-200-isvc-78590-predictor-599b7b7f46-8r44l\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") " pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.467725 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.467688    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.603157 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.603112    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"]
Jul 02 14:16:06.605013 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:16:06.604991    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a1816a_ea79_42c5_af42_a3f922f1526e.slice/crio-f5921f72a1eacaabcc0b5aa92e5b094245c6c22d81c6243907627982ab5e8fed WatchSource:0}: Error finding container f5921f72a1eacaabcc0b5aa92e5b094245c6c22d81c6243907627982ab5e8fed: Status 404 returned error can't find the container with id f5921f72a1eacaabcc0b5aa92e5b094245c6c22d81c6243907627982ab5e8fed
Jul 02 14:16:06.746281 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.746241    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" event={"ID":"24a1816a-ea79-42c5-af42-a3f922f1526e","Type":"ContainerStarted","Data":"36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e"}
Jul 02 14:16:06.746465 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.746441    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.746538 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.746491    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:06.746538 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.746519    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" event={"ID":"24a1816a-ea79-42c5-af42-a3f922f1526e","Type":"ContainerStarted","Data":"e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247"}
Jul 02 14:16:06.746652 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.746551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" event={"ID":"24a1816a-ea79-42c5-af42-a3f922f1526e","Type":"ContainerStarted","Data":"f5921f72a1eacaabcc0b5aa92e5b094245c6c22d81c6243907627982ab5e8fed"}
Jul 02 14:16:06.747868 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.747846    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 02 14:16:06.748105 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.748087    2574 generic.go:356] "Generic (PLEG): container finished" podID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerID="76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371" exitCode=2
Jul 02 14:16:06.748196 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.748156    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" event={"ID":"4ae79875-c7f8-4433-ba70-8d2d76b10627","Type":"ContainerDied","Data":"76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371"}
Jul 02 14:16:06.768312 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:06.768231    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" podStartSLOduration=0.768217464 podStartE2EDuration="768.217464ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:06.766782682 +0000 UTC m=+1903.962406188" watchObservedRunningTime="2026-07-02 14:16:06.768217464 +0000 UTC m=+1903.963840972"
Jul 02 14:16:07.699997 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:07.699952    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" podUID="9147221c-2bcd-496d-a48f-e8cd49036e4d" containerName="splitter-graph-09121" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:16:07.752126 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:07.752083    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 02 14:16:09.381269 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.381242    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:16:09.428608 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.428578    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7z4k\" (UniqueName: \"kubernetes.io/projected/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-api-access-q7z4k\") pod \"4ae79875-c7f8-4433-ba70-8d2d76b10627\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") "
Jul 02 14:16:09.428789 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.428635    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae79875-c7f8-4433-ba70-8d2d76b10627-proxy-tls\") pod \"4ae79875-c7f8-4433-ba70-8d2d76b10627\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") "
Jul 02 14:16:09.428789 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.428712    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-rbac-proxy-sar-config\") pod \"4ae79875-c7f8-4433-ba70-8d2d76b10627\" (UID: \"4ae79875-c7f8-4433-ba70-8d2d76b10627\") "
Jul 02 14:16:09.430197 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.430170    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ae79875-c7f8-4433-ba70-8d2d76b10627" (UID: "4ae79875-c7f8-4433-ba70-8d2d76b10627"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:16:09.431031 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.431005    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-api-access-q7z4k" (OuterVolumeSpecName: "kube-api-access-q7z4k") pod "4ae79875-c7f8-4433-ba70-8d2d76b10627" (UID: "4ae79875-c7f8-4433-ba70-8d2d76b10627"). InnerVolumeSpecName "kube-api-access-q7z4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:16:09.431266 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.431245    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae79875-c7f8-4433-ba70-8d2d76b10627-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ae79875-c7f8-4433-ba70-8d2d76b10627" (UID: "4ae79875-c7f8-4433-ba70-8d2d76b10627"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:16:09.505926 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.505842    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" podUID="8d9d8706-f755-42c1-9fb7-8c31e211345d" containerName="sequence-graph-6d218" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:16:09.530354 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.530320    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:16:09.530354 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.530352    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7z4k\" (UniqueName: \"kubernetes.io/projected/4ae79875-c7f8-4433-ba70-8d2d76b10627-kube-api-access-q7z4k\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:16:09.530530 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.530367    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae79875-c7f8-4433-ba70-8d2d76b10627-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:16:09.686880 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.686839    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 14:16:09.762710 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.762625    2574 generic.go:356] "Generic (PLEG): container finished" podID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerID="5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768" exitCode=0
Jul 02 14:16:09.762710 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.762681    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"
Jul 02 14:16:09.762710 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.762681    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" event={"ID":"4ae79875-c7f8-4433-ba70-8d2d76b10627","Type":"ContainerDied","Data":"5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768"}
Jul 02 14:16:09.762942 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.762742    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm" event={"ID":"4ae79875-c7f8-4433-ba70-8d2d76b10627","Type":"ContainerDied","Data":"927bca7a518c63df1bb9b19fd30e4d7c4be1565d012659626c46195461b80a5d"}
Jul 02 14:16:09.762942 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.762767    2574 scope.go:117] "RemoveContainer" containerID="76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371"
Jul 02 14:16:09.771822 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.771804    2574 scope.go:117] "RemoveContainer" containerID="5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768"
Jul 02 14:16:09.779115 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.779095    2574 scope.go:117] "RemoveContainer" containerID="76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371"
Jul 02 14:16:09.779400 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:16:09.779381    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371\": container with ID starting with 76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371 not found: ID does not exist" containerID="76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371"
Jul 02 14:16:09.779447 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.779410    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371"} err="failed to get container status \"76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371\": rpc error: code = NotFound desc = could not find container \"76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371\": container with ID starting with 76a9d7460b1c3f4e0f3414c4eb21ec99a72c27f7121e779980b46ab9fdbec371 not found: ID does not exist"
Jul 02 14:16:09.779447 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.779427    2574 scope.go:117] "RemoveContainer" containerID="5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768"
Jul 02 14:16:09.779675 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:16:09.779658    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768\": container with ID starting with 5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768 not found: ID does not exist" containerID="5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768"
Jul 02 14:16:09.779715 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.779681    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768"} err="failed to get container status \"5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768\": rpc error: code = NotFound desc = could not find container \"5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768\": container with ID starting with 5d2592e06ef031c453606cfe83da7df587806867c2a2f58e65a82a2ab53c1768 not found: ID does not exist"
Jul 02 14:16:09.785192 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.785171    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"]
Jul 02 14:16:09.787249 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:09.787228    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09121-predictor-66bcd6b46-k2xnm"]
Jul 02 14:16:11.350956 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:11.350925    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" path="/var/lib/kubelet/pods/4ae79875-c7f8-4433-ba70-8d2d76b10627/volumes"
Jul 02 14:16:12.699324 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:12.699288    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" podUID="9147221c-2bcd-496d-a48f-e8cd49036e4d" containerName="splitter-graph-09121" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:16:12.756851 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:12.756823    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:12.757333 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:12.757308    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 02 14:16:14.506055 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:14.506011    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" podUID="8d9d8706-f755-42c1-9fb7-8c31e211345d" containerName="sequence-graph-6d218" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:16:17.700186 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:17.700115    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" podUID="9147221c-2bcd-496d-a48f-e8cd49036e4d" containerName="splitter-graph-09121" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:16:17.700647 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:17.700279    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:16:19.505978 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:19.505934    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" podUID="8d9d8706-f755-42c1-9fb7-8c31e211345d" containerName="sequence-graph-6d218" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:16:19.685830 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:19.685791    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 14:16:21.824431 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:21.824398    2574 generic.go:356] "Generic (PLEG): container finished" podID="8d9d8706-f755-42c1-9fb7-8c31e211345d" containerID="659c025279176840838bd5839e125a8fb9fa4a9c46e084748135ec64ddc5c011" exitCode=0
Jul 02 14:16:21.824781 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:21.824464    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" event={"ID":"8d9d8706-f755-42c1-9fb7-8c31e211345d","Type":"ContainerDied","Data":"659c025279176840838bd5839e125a8fb9fa4a9c46e084748135ec64ddc5c011"}
Jul 02 14:16:22.328004 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.327979    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:16:22.444058 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.443968    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9d8706-f755-42c1-9fb7-8c31e211345d-openshift-service-ca-bundle\") pod \"8d9d8706-f755-42c1-9fb7-8c31e211345d\" (UID: \"8d9d8706-f755-42c1-9fb7-8c31e211345d\") "
Jul 02 14:16:22.444239 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.444061    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9d8706-f755-42c1-9fb7-8c31e211345d-proxy-tls\") pod \"8d9d8706-f755-42c1-9fb7-8c31e211345d\" (UID: \"8d9d8706-f755-42c1-9fb7-8c31e211345d\") "
Jul 02 14:16:22.444415 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.444387    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9d8706-f755-42c1-9fb7-8c31e211345d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8d9d8706-f755-42c1-9fb7-8c31e211345d" (UID: "8d9d8706-f755-42c1-9fb7-8c31e211345d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:16:22.446029 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.446011    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9d8706-f755-42c1-9fb7-8c31e211345d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d9d8706-f755-42c1-9fb7-8c31e211345d" (UID: "8d9d8706-f755-42c1-9fb7-8c31e211345d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:16:22.544766 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.544733    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d9d8706-f755-42c1-9fb7-8c31e211345d-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:16:22.544766 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.544759    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9d8706-f755-42c1-9fb7-8c31e211345d-openshift-service-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:16:22.700286 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.700185    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" podUID="9147221c-2bcd-496d-a48f-e8cd49036e4d" containerName="splitter-graph-09121" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:16:22.757359 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.757321    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 02 14:16:22.830715 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.830689    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"
Jul 02 14:16:22.831206 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.830729    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45" event={"ID":"8d9d8706-f755-42c1-9fb7-8c31e211345d","Type":"ContainerDied","Data":"ce181cc3a24d9661183c36ef23216353c939721f38806bd9651739f39da0c19e"}
Jul 02 14:16:22.831206 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.830796    2574 scope.go:117] "RemoveContainer" containerID="659c025279176840838bd5839e125a8fb9fa4a9c46e084748135ec64ddc5c011"
Jul 02 14:16:22.853345 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.853318    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"]
Jul 02 14:16:22.857380 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:22.857360    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6d218-57587c6648-hpj45"]
Jul 02 14:16:23.351005 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:23.350964    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d9d8706-f755-42c1-9fb7-8c31e211345d" path="/var/lib/kubelet/pods/8d9d8706-f755-42c1-9fb7-8c31e211345d/volumes"
Jul 02 14:16:27.700081 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:27.700044    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" podUID="9147221c-2bcd-496d-a48f-e8cd49036e4d" containerName="splitter-graph-09121" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:16:29.686328 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:29.686287    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 14:16:32.699887 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:32.699847    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" podUID="9147221c-2bcd-496d-a48f-e8cd49036e4d" containerName="splitter-graph-09121" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:16:32.757976 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:32.757941    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 02 14:16:36.029347 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:16:36.029315    2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9147221c_2bcd_496d_a48f_e8cd49036e4d.slice/crio-conmon-ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063.scope\": RecentStats: unable to find data in memory cache]"
Jul 02 14:16:36.154373 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.154349    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:16:36.263710 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.263619    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9147221c-2bcd-496d-a48f-e8cd49036e4d-proxy-tls\") pod \"9147221c-2bcd-496d-a48f-e8cd49036e4d\" (UID: \"9147221c-2bcd-496d-a48f-e8cd49036e4d\") "
Jul 02 14:16:36.263880 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.263720    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147221c-2bcd-496d-a48f-e8cd49036e4d-openshift-service-ca-bundle\") pod \"9147221c-2bcd-496d-a48f-e8cd49036e4d\" (UID: \"9147221c-2bcd-496d-a48f-e8cd49036e4d\") "
Jul 02 14:16:36.264080 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.264055    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9147221c-2bcd-496d-a48f-e8cd49036e4d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9147221c-2bcd-496d-a48f-e8cd49036e4d" (UID: "9147221c-2bcd-496d-a48f-e8cd49036e4d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:16:36.265694 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.265676    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9147221c-2bcd-496d-a48f-e8cd49036e4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9147221c-2bcd-496d-a48f-e8cd49036e4d" (UID: "9147221c-2bcd-496d-a48f-e8cd49036e4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:16:36.364443 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.364408    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9147221c-2bcd-496d-a48f-e8cd49036e4d-openshift-service-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:16:36.364443 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.364437    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9147221c-2bcd-496d-a48f-e8cd49036e4d-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:16:36.893355 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.893321    2574 generic.go:356] "Generic (PLEG): container finished" podID="9147221c-2bcd-496d-a48f-e8cd49036e4d" containerID="ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063" exitCode=0
Jul 02 14:16:36.893617 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.893365    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"
Jul 02 14:16:36.893617 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.893409    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" event={"ID":"9147221c-2bcd-496d-a48f-e8cd49036e4d","Type":"ContainerDied","Data":"ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063"}
Jul 02 14:16:36.893617 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.893464    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns" event={"ID":"9147221c-2bcd-496d-a48f-e8cd49036e4d","Type":"ContainerDied","Data":"5358c4174c10a34727d73e362ab15ad014f31ad91b7b301759d4b190da66ff28"}
Jul 02 14:16:36.893617 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.893489    2574 scope.go:117] "RemoveContainer" containerID="ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063"
Jul 02 14:16:36.902358 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.902339    2574 scope.go:117] "RemoveContainer" containerID="ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063"
Jul 02 14:16:36.902569 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:16:36.902554    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063\": container with ID starting with ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063 not found: ID does not exist" containerID="ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063"
Jul 02 14:16:36.902611 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.902576    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063"} err="failed to get container status \"ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063\": rpc error: code = NotFound desc = could not find container \"ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063\": container with ID starting with ae6401bc9d8295922039ab74a47e569d0e1a5da3b6353c737337d408cc741063 not found: ID does not exist"
Jul 02 14:16:36.915390 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.915363    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"]
Jul 02 14:16:36.918916 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:36.918896    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-09121-864f98c7fd-rj8ns"]
Jul 02 14:16:37.350574 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:37.350540    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9147221c-2bcd-496d-a48f-e8cd49036e4d" path="/var/lib/kubelet/pods/9147221c-2bcd-496d-a48f-e8cd49036e4d/volumes"
Jul 02 14:16:39.686807 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:39.686776    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:16:42.757495 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:42.757455    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 02 14:16:51.938423 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938387    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"]
Jul 02 14:16:51.938797 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938722    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9147221c-2bcd-496d-a48f-e8cd49036e4d" containerName="splitter-graph-09121"
Jul 02 14:16:51.938797 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938734    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9147221c-2bcd-496d-a48f-e8cd49036e4d" containerName="splitter-graph-09121"
Jul 02 14:16:51.938797 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938744    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kserve-container"
Jul 02 14:16:51.938797 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938750    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kserve-container"
Jul 02 14:16:51.938797 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938766    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kube-rbac-proxy"
Jul 02 14:16:51.938797 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938772    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kube-rbac-proxy"
Jul 02 14:16:51.938797 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938783    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9d8706-f755-42c1-9fb7-8c31e211345d" containerName="sequence-graph-6d218"
Jul 02 14:16:51.938797 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938788    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9d8706-f755-42c1-9fb7-8c31e211345d" containerName="sequence-graph-6d218"
Jul 02 14:16:51.939051 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938835    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9147221c-2bcd-496d-a48f-e8cd49036e4d" containerName="splitter-graph-09121"
Jul 02 14:16:51.939051 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938841    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9d8706-f755-42c1-9fb7-8c31e211345d" containerName="sequence-graph-6d218"
Jul 02 14:16:51.939051 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938852    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kube-rbac-proxy"
Jul 02 14:16:51.939051 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.938858    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae79875-c7f8-4433-ba70-8d2d76b10627" containerName="kserve-container"
Jul 02 14:16:51.943040 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.943020    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:16:51.946492 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.946472    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8d2ee-serving-cert\""
Jul 02 14:16:51.946608 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.946585    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8d2ee-kube-rbac-proxy-sar-config\""
Jul 02 14:16:51.959242 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:51.959216    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"]
Jul 02 14:16:52.097509 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:52.097467    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f159dcf8-4505-42e0-86ce-22e768bdf8a6-proxy-tls\") pod \"switch-graph-8d2ee-545b848957-x45gp\" (UID: \"f159dcf8-4505-42e0-86ce-22e768bdf8a6\") " pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:16:52.097688 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:52.097520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f159dcf8-4505-42e0-86ce-22e768bdf8a6-openshift-service-ca-bundle\") pod \"switch-graph-8d2ee-545b848957-x45gp\" (UID: \"f159dcf8-4505-42e0-86ce-22e768bdf8a6\") " pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:16:52.198683 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:52.198592    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f159dcf8-4505-42e0-86ce-22e768bdf8a6-proxy-tls\") pod \"switch-graph-8d2ee-545b848957-x45gp\" (UID: \"f159dcf8-4505-42e0-86ce-22e768bdf8a6\") " pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:16:52.198683 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:52.198631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f159dcf8-4505-42e0-86ce-22e768bdf8a6-openshift-service-ca-bundle\") pod \"switch-graph-8d2ee-545b848957-x45gp\" (UID: \"f159dcf8-4505-42e0-86ce-22e768bdf8a6\") " pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:16:52.198964 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:16:52.198727    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-8d2ee-serving-cert: secret "switch-graph-8d2ee-serving-cert" not found
Jul 02 14:16:52.198964 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:16:52.198796    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f159dcf8-4505-42e0-86ce-22e768bdf8a6-proxy-tls podName:f159dcf8-4505-42e0-86ce-22e768bdf8a6 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:52.698781173 +0000 UTC m=+1949.894404658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f159dcf8-4505-42e0-86ce-22e768bdf8a6-proxy-tls") pod "switch-graph-8d2ee-545b848957-x45gp" (UID: "f159dcf8-4505-42e0-86ce-22e768bdf8a6") : secret "switch-graph-8d2ee-serving-cert" not found
Jul 02 14:16:52.199379 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:52.199357    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f159dcf8-4505-42e0-86ce-22e768bdf8a6-openshift-service-ca-bundle\") pod \"switch-graph-8d2ee-545b848957-x45gp\" (UID: \"f159dcf8-4505-42e0-86ce-22e768bdf8a6\") " pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:16:52.703185 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:52.703130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f159dcf8-4505-42e0-86ce-22e768bdf8a6-proxy-tls\") pod \"switch-graph-8d2ee-545b848957-x45gp\" (UID: \"f159dcf8-4505-42e0-86ce-22e768bdf8a6\") " pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:16:52.705528 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:52.705497    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f159dcf8-4505-42e0-86ce-22e768bdf8a6-proxy-tls\") pod \"switch-graph-8d2ee-545b848957-x45gp\" (UID: \"f159dcf8-4505-42e0-86ce-22e768bdf8a6\") " pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:16:52.758323 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:52.758297    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:16:52.852851 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:52.852821    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:16:52.982278 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:52.982259    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"]
Jul 02 14:16:52.984509 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:16:52.984472    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf159dcf8_4505_42e0_86ce_22e768bdf8a6.slice/crio-9bcb1891aa503fd34c10fbb92471b4821e72d37ff9acba0a985fa8a20e3d38b3 WatchSource:0}: Error finding container 9bcb1891aa503fd34c10fbb92471b4821e72d37ff9acba0a985fa8a20e3d38b3: Status 404 returned error can't find the container with id 9bcb1891aa503fd34c10fbb92471b4821e72d37ff9acba0a985fa8a20e3d38b3
Jul 02 14:16:53.971479 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:53.971445    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" event={"ID":"f159dcf8-4505-42e0-86ce-22e768bdf8a6","Type":"ContainerStarted","Data":"bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6"}
Jul 02 14:16:53.971667 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:53.971498    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" event={"ID":"f159dcf8-4505-42e0-86ce-22e768bdf8a6","Type":"ContainerStarted","Data":"9bcb1891aa503fd34c10fbb92471b4821e72d37ff9acba0a985fa8a20e3d38b3"}
Jul 02 14:16:53.971667 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:53.971599    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:16:53.991664 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:53.991621    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" podStartSLOduration=2.9916093630000002 podStartE2EDuration="2.991609363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:53.991298465 +0000 UTC m=+1951.186921971" watchObservedRunningTime="2026-07-02 14:16:53.991609363 +0000 UTC m=+1951.187232869"
Jul 02 14:16:59.981204 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:16:59.981171    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:17:06.209315 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.209286    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"]
Jul 02 14:17:06.212290 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.212273    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:17:06.215199 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.215178    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-78590-kube-rbac-proxy-sar-config\""
Jul 02 14:17:06.215321 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.215205    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-78590-serving-cert\""
Jul 02 14:17:06.222500 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.222478    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"]
Jul 02 14:17:06.302988 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.302958    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6acd2f91-4605-4cfb-87a4-a3828dd4d960-proxy-tls\") pod \"splitter-graph-78590-85df59669f-6j9xq\" (UID: \"6acd2f91-4605-4cfb-87a4-a3828dd4d960\") " pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:17:06.303194 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.303020    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acd2f91-4605-4cfb-87a4-a3828dd4d960-openshift-service-ca-bundle\") pod \"splitter-graph-78590-85df59669f-6j9xq\" (UID: \"6acd2f91-4605-4cfb-87a4-a3828dd4d960\") " pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:17:06.404198 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.404165    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6acd2f91-4605-4cfb-87a4-a3828dd4d960-proxy-tls\") pod \"splitter-graph-78590-85df59669f-6j9xq\" (UID: \"6acd2f91-4605-4cfb-87a4-a3828dd4d960\") " pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:17:06.404383 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.404216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acd2f91-4605-4cfb-87a4-a3828dd4d960-openshift-service-ca-bundle\") pod \"splitter-graph-78590-85df59669f-6j9xq\" (UID: \"6acd2f91-4605-4cfb-87a4-a3828dd4d960\") " pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:17:06.404383 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:17:06.404326    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-78590-serving-cert: secret "splitter-graph-78590-serving-cert" not found
Jul 02 14:17:06.404501 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:17:06.404432    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acd2f91-4605-4cfb-87a4-a3828dd4d960-proxy-tls podName:6acd2f91-4605-4cfb-87a4-a3828dd4d960 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:06.904409134 +0000 UTC m=+1964.100032636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6acd2f91-4605-4cfb-87a4-a3828dd4d960-proxy-tls") pod "splitter-graph-78590-85df59669f-6j9xq" (UID: "6acd2f91-4605-4cfb-87a4-a3828dd4d960") : secret "splitter-graph-78590-serving-cert" not found
Jul 02 14:17:06.404769 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.404752    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acd2f91-4605-4cfb-87a4-a3828dd4d960-openshift-service-ca-bundle\") pod \"splitter-graph-78590-85df59669f-6j9xq\" (UID: \"6acd2f91-4605-4cfb-87a4-a3828dd4d960\") " pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:17:06.908459 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.908428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6acd2f91-4605-4cfb-87a4-a3828dd4d960-proxy-tls\") pod \"splitter-graph-78590-85df59669f-6j9xq\" (UID: \"6acd2f91-4605-4cfb-87a4-a3828dd4d960\") " pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:17:06.910782 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:06.910760    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6acd2f91-4605-4cfb-87a4-a3828dd4d960-proxy-tls\") pod \"splitter-graph-78590-85df59669f-6j9xq\" (UID: \"6acd2f91-4605-4cfb-87a4-a3828dd4d960\") " pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:17:07.122463 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:07.122431    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:17:07.258958 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:07.258931    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"]
Jul 02 14:17:07.260593 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:17:07.260569    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acd2f91_4605_4cfb_87a4_a3828dd4d960.slice/crio-825d9eacce30b9da59ad30d760289eaaabf304a36eec47a47ca7bf6e829b1eeb WatchSource:0}: Error finding container 825d9eacce30b9da59ad30d760289eaaabf304a36eec47a47ca7bf6e829b1eeb: Status 404 returned error can't find the container with id 825d9eacce30b9da59ad30d760289eaaabf304a36eec47a47ca7bf6e829b1eeb
Jul 02 14:17:08.035081 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:08.035047    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" event={"ID":"6acd2f91-4605-4cfb-87a4-a3828dd4d960","Type":"ContainerStarted","Data":"1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12"}
Jul 02 14:17:08.035289 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:08.035153    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" event={"ID":"6acd2f91-4605-4cfb-87a4-a3828dd4d960","Type":"ContainerStarted","Data":"825d9eacce30b9da59ad30d760289eaaabf304a36eec47a47ca7bf6e829b1eeb"}
Jul 02 14:17:08.035289 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:08.035191    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:17:08.055441 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:08.055395    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" podStartSLOduration=2.055381916 podStartE2EDuration="2.055381916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:17:08.054469531 +0000 UTC m=+1965.250093038" watchObservedRunningTime="2026-07-02 14:17:08.055381916 +0000 UTC m=+1965.251005450"
Jul 02 14:17:14.044898 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:17:14.044870    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:19:23.439735 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:19:23.439705    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:19:23.445588 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:19:23.445567    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:19:23.446379 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:19:23.446361    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:19:23.451779 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:19:23.451763    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:24:23.469265 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:24:23.469159    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:24:23.475181 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:24:23.475160    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:24:23.475429 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:24:23.475413    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:24:23.480607 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:24:23.480591    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:25:20.830258 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:20.830224    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"]
Jul 02 14:25:20.830750 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:20.830409    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" podUID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" containerName="splitter-graph-78590" containerID="cri-o://1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12" gracePeriod=30
Jul 02 14:25:20.957319 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:20.957284    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"]
Jul 02 14:25:20.957561 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:20.957518    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kserve-container" containerID="cri-o://e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247" gracePeriod=30
Jul 02 14:25:20.957627 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:20.957564    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kube-rbac-proxy" containerID="cri-o://36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e" gracePeriod=30
Jul 02 14:25:21.194101 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:21.194070    2574 generic.go:356] "Generic (PLEG): container finished" podID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerID="36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e" exitCode=2
Jul 02 14:25:21.194295 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:21.194160    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" event={"ID":"24a1816a-ea79-42c5-af42-a3f922f1526e","Type":"ContainerDied","Data":"36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e"}
Jul 02 14:25:22.752465 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:22.752424    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 02 14:25:22.757387 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:22.757358    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 02 14:25:24.007905 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.007880    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:25:24.045389 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.045344    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" podUID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" containerName="splitter-graph-78590" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:25:24.140792 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.140713    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-rbac-proxy-sar-config\") pod \"24a1816a-ea79-42c5-af42-a3f922f1526e\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") "
Jul 02 14:25:24.140792 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.140791    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9vpk\" (UniqueName: \"kubernetes.io/projected/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-api-access-b9vpk\") pod \"24a1816a-ea79-42c5-af42-a3f922f1526e\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") "
Jul 02 14:25:24.141003 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.140872    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a1816a-ea79-42c5-af42-a3f922f1526e-proxy-tls\") pod \"24a1816a-ea79-42c5-af42-a3f922f1526e\" (UID: \"24a1816a-ea79-42c5-af42-a3f922f1526e\") "
Jul 02 14:25:24.141177 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.141130    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "24a1816a-ea79-42c5-af42-a3f922f1526e" (UID: "24a1816a-ea79-42c5-af42-a3f922f1526e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:25:24.142820 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.142800    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a1816a-ea79-42c5-af42-a3f922f1526e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24a1816a-ea79-42c5-af42-a3f922f1526e" (UID: "24a1816a-ea79-42c5-af42-a3f922f1526e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:25:24.142873 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.142841    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-api-access-b9vpk" (OuterVolumeSpecName: "kube-api-access-b9vpk") pod "24a1816a-ea79-42c5-af42-a3f922f1526e" (UID: "24a1816a-ea79-42c5-af42-a3f922f1526e"). InnerVolumeSpecName "kube-api-access-b9vpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:25:24.208730 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.208701    2574 generic.go:356] "Generic (PLEG): container finished" podID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerID="e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247" exitCode=0
Jul 02 14:25:24.208896 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.208756    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"
Jul 02 14:25:24.208896 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.208780    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" event={"ID":"24a1816a-ea79-42c5-af42-a3f922f1526e","Type":"ContainerDied","Data":"e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247"}
Jul 02 14:25:24.208896 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.208840    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l" event={"ID":"24a1816a-ea79-42c5-af42-a3f922f1526e","Type":"ContainerDied","Data":"f5921f72a1eacaabcc0b5aa92e5b094245c6c22d81c6243907627982ab5e8fed"}
Jul 02 14:25:24.208896 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.208867    2574 scope.go:117] "RemoveContainer" containerID="36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e"
Jul 02 14:25:24.217779 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.217766    2574 scope.go:117] "RemoveContainer" containerID="e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247"
Jul 02 14:25:24.226721 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.226699    2574 scope.go:117] "RemoveContainer" containerID="36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e"
Jul 02 14:25:24.226991 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:25:24.226973    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e\": container with ID starting with 36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e not found: ID does not exist" containerID="36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e"
Jul 02 14:25:24.227071 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.226997    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e"} err="failed to get container status \"36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e\": rpc error: code = NotFound desc = could not find container \"36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e\": container with ID starting with 36d7aa68d3d350c871af52d657056fc5e6290070f85fa3f875fb9155fa7fc33e not found: ID does not exist"
Jul 02 14:25:24.227071 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.227015    2574 scope.go:117] "RemoveContainer" containerID="e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247"
Jul 02 14:25:24.227277 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:25:24.227260    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247\": container with ID starting with e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247 not found: ID does not exist" containerID="e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247"
Jul 02 14:25:24.227325 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.227280    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247"} err="failed to get container status \"e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247\": rpc error: code = NotFound desc = could not find container \"e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247\": container with ID starting with e80440b5554749a8e6a4617c636ab426e3893c46d291924dc0bf378f29011247 not found: ID does not exist"
Jul 02 14:25:24.230943 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.230917    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"]
Jul 02 14:25:24.234736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.234718    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78590-predictor-599b7b7f46-8r44l"]
Jul 02 14:25:24.242378 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.242362    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:25:24.242445 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.242382    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9vpk\" (UniqueName: \"kubernetes.io/projected/24a1816a-ea79-42c5-af42-a3f922f1526e-kube-api-access-b9vpk\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:25:24.242445 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:24.242393    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a1816a-ea79-42c5-af42-a3f922f1526e-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:25:25.350711 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:25.350674    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" path="/var/lib/kubelet/pods/24a1816a-ea79-42c5-af42-a3f922f1526e/volumes"
Jul 02 14:25:29.044279 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:29.044242    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" podUID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" containerName="splitter-graph-78590" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:25:34.044450 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:34.044413    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" podUID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" containerName="splitter-graph-78590" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:25:34.044832 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:34.044517    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:25:39.044282 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:39.044244    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" podUID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" containerName="splitter-graph-78590" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:25:44.044007 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:44.043973    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" podUID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" containerName="splitter-graph-78590" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:25:49.043922 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:49.043884    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" podUID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" containerName="splitter-graph-78590" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:25:50.982580 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:50.982558    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:25:51.067706 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.067672    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acd2f91-4605-4cfb-87a4-a3828dd4d960-openshift-service-ca-bundle\") pod \"6acd2f91-4605-4cfb-87a4-a3828dd4d960\" (UID: \"6acd2f91-4605-4cfb-87a4-a3828dd4d960\") "
Jul 02 14:25:51.067866 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.067742    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6acd2f91-4605-4cfb-87a4-a3828dd4d960-proxy-tls\") pod \"6acd2f91-4605-4cfb-87a4-a3828dd4d960\" (UID: \"6acd2f91-4605-4cfb-87a4-a3828dd4d960\") "
Jul 02 14:25:51.068174 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.068118    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6acd2f91-4605-4cfb-87a4-a3828dd4d960-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6acd2f91-4605-4cfb-87a4-a3828dd4d960" (UID: "6acd2f91-4605-4cfb-87a4-a3828dd4d960"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:25:51.069828 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.069802    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6acd2f91-4605-4cfb-87a4-a3828dd4d960-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6acd2f91-4605-4cfb-87a4-a3828dd4d960" (UID: "6acd2f91-4605-4cfb-87a4-a3828dd4d960"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:25:51.169151 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.169107    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6acd2f91-4605-4cfb-87a4-a3828dd4d960-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:25:51.169300 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.169156    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6acd2f91-4605-4cfb-87a4-a3828dd4d960-openshift-service-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:25:51.331225 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.331190    2574 generic.go:356] "Generic (PLEG): container finished" podID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" containerID="1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12" exitCode=0
Jul 02 14:25:51.331409 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.331235    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"
Jul 02 14:25:51.331409 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.331277    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" event={"ID":"6acd2f91-4605-4cfb-87a4-a3828dd4d960","Type":"ContainerDied","Data":"1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12"}
Jul 02 14:25:51.331409 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.331331    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq" event={"ID":"6acd2f91-4605-4cfb-87a4-a3828dd4d960","Type":"ContainerDied","Data":"825d9eacce30b9da59ad30d760289eaaabf304a36eec47a47ca7bf6e829b1eeb"}
Jul 02 14:25:51.331409 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.331357    2574 scope.go:117] "RemoveContainer" containerID="1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12"
Jul 02 14:25:51.339734 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.339719    2574 scope.go:117] "RemoveContainer" containerID="1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12"
Jul 02 14:25:51.339951 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:25:51.339929    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12\": container with ID starting with 1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12 not found: ID does not exist" containerID="1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12"
Jul 02 14:25:51.340018 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.339963    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12"} err="failed to get container status \"1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12\": rpc error: code = NotFound desc = could not find container \"1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12\": container with ID starting with 1d7d480821b86c6fca7c02acc817cd12dfcc4daeefe9471baf28195d5b794c12 not found: ID does not exist"
Jul 02 14:25:51.351696 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.351673    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"]
Jul 02 14:25:51.355943 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:51.355924    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-78590-85df59669f-6j9xq"]
Jul 02 14:25:53.351872 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:25:53.351826    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" path="/var/lib/kubelet/pods/6acd2f91-4605-4cfb-87a4-a3828dd4d960/volumes"
Jul 02 14:29:23.494625 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:29:23.494514    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:29:23.500328 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:29:23.500306    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:29:23.500718 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:29:23.500697    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:29:23.506951 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:29:23.506934    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:33:11.420914 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:11.420876    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"]
Jul 02 14:33:11.421472 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:11.421082    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" podUID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" containerName="switch-graph-8d2ee" containerID="cri-o://bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6" gracePeriod=30
Jul 02 14:33:11.610554 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:11.610521    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"]
Jul 02 14:33:11.610806 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:11.610765    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kserve-container" containerID="cri-o://ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff" gracePeriod=30
Jul 02 14:33:11.610900 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:11.610839    2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kube-rbac-proxy" containerID="cri-o://5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db" gracePeriod=30
Jul 02 14:33:12.241861 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:12.241823    2574 generic.go:356] "Generic (PLEG): container finished" podID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerID="5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db" exitCode=2
Jul 02 14:33:12.242028 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:12.241898    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" event={"ID":"f294d0b1-01b6-40de-aef0-c6cdb40552a3","Type":"ContainerDied","Data":"5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db"}
Jul 02 14:33:14.660910 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:14.660885    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:33:14.681323 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:14.681302    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f294d0b1-01b6-40de-aef0-c6cdb40552a3-proxy-tls\") pod \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") "
Jul 02 14:33:14.681440 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:14.681349    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-rbac-proxy-sar-config\") pod \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") "
Jul 02 14:33:14.681440 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:14.681428    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkb6w\" (UniqueName: \"kubernetes.io/projected/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-api-access-vkb6w\") pod \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\" (UID: \"f294d0b1-01b6-40de-aef0-c6cdb40552a3\") "
Jul 02 14:33:14.681673 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:14.681653    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f294d0b1-01b6-40de-aef0-c6cdb40552a3" (UID: "f294d0b1-01b6-40de-aef0-c6cdb40552a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:33:14.683207 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:14.683182    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f294d0b1-01b6-40de-aef0-c6cdb40552a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f294d0b1-01b6-40de-aef0-c6cdb40552a3" (UID: "f294d0b1-01b6-40de-aef0-c6cdb40552a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:33:14.683373 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:14.683347    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-api-access-vkb6w" (OuterVolumeSpecName: "kube-api-access-vkb6w") pod "f294d0b1-01b6-40de-aef0-c6cdb40552a3" (UID: "f294d0b1-01b6-40de-aef0-c6cdb40552a3"). InnerVolumeSpecName "kube-api-access-vkb6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:33:14.781913 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:14.781883    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:33:14.781913 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:14.781908    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkb6w\" (UniqueName: \"kubernetes.io/projected/f294d0b1-01b6-40de-aef0-c6cdb40552a3-kube-api-access-vkb6w\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:33:14.781913 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:14.781918    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f294d0b1-01b6-40de-aef0-c6cdb40552a3-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:33:14.979744 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:14.979653    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" podUID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" containerName="switch-graph-8d2ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:33:15.256596 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.256514    2574 generic.go:356] "Generic (PLEG): container finished" podID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerID="ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff" exitCode=0
Jul 02 14:33:15.256596 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.256567    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"
Jul 02 14:33:15.256596 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.256584    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" event={"ID":"f294d0b1-01b6-40de-aef0-c6cdb40552a3","Type":"ContainerDied","Data":"ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff"}
Jul 02 14:33:15.256825 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.256634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8" event={"ID":"f294d0b1-01b6-40de-aef0-c6cdb40552a3","Type":"ContainerDied","Data":"d53b677923a884e15a9c0a093cb8d7db4a7603fa326e9b50b3ff9a59a8be116a"}
Jul 02 14:33:15.256825 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.256659    2574 scope.go:117] "RemoveContainer" containerID="5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db"
Jul 02 14:33:15.265615 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.265598    2574 scope.go:117] "RemoveContainer" containerID="ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff"
Jul 02 14:33:15.272692 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.272678    2574 scope.go:117] "RemoveContainer" containerID="5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db"
Jul 02 14:33:15.272916 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:33:15.272899    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db\": container with ID starting with 5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db not found: ID does not exist" containerID="5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db"
Jul 02 14:33:15.272973 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.272923    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db"} err="failed to get container status \"5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db\": rpc error: code = NotFound desc = could not find container \"5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db\": container with ID starting with 5b4e5ddc0e3654cf15a218e49a685a729716311d6e86491ccc0469fa666c54db not found: ID does not exist"
Jul 02 14:33:15.272973 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.272950    2574 scope.go:117] "RemoveContainer" containerID="ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff"
Jul 02 14:33:15.273228 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:33:15.273206    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff\": container with ID starting with ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff not found: ID does not exist" containerID="ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff"
Jul 02 14:33:15.273299 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.273238    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff"} err="failed to get container status \"ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff\": rpc error: code = NotFound desc = could not find container \"ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff\": container with ID starting with ef6455b18a6f9be1fb752389018a44e1ac23b11abbdbfba6679f5673246b3aff not found: ID does not exist"
Jul 02 14:33:15.291690 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.291670    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"]
Jul 02 14:33:15.298464 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.298445    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d2ee-predictor-5d9c964d96-7txl8"]
Jul 02 14:33:15.350052 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:15.350029    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" path="/var/lib/kubelet/pods/f294d0b1-01b6-40de-aef0-c6cdb40552a3/volumes"
Jul 02 14:33:19.980153 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:19.980098    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" podUID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" containerName="switch-graph-8d2ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:33:24.980163 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:24.980102    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" podUID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" containerName="switch-graph-8d2ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:33:24.980634 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:24.980233    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:33:28.429954 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:28.429924    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:29.370319 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:29.370292    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:29.979917 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:29.979883    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" podUID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" containerName="switch-graph-8d2ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:33:30.280125 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:30.280043    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:31.139841 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:31.139809    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:32.004600 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:32.004570    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:32.874277 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:32.874247    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:33.702361 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:33.702335    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:34.577529 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:34.577503    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:34.979839 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:34.979800    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" podUID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" containerName="switch-graph-8d2ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:33:35.508268 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:35.508230    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:36.365881 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:36.365856    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:37.240253 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:37.240214    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:38.100758 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:38.100732    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:38.991067 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:38.991036    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:39.884650 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:39.884621    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-8d2ee-545b848957-x45gp_f159dcf8-4505-42e0-86ce-22e768bdf8a6/switch-graph-8d2ee/0.log"
Jul 02 14:33:39.980324 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:39.980288    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" podUID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" containerName="switch-graph-8d2ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:33:41.570204 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:41.570181    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:33:41.700748 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:41.700659    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f159dcf8-4505-42e0-86ce-22e768bdf8a6-openshift-service-ca-bundle\") pod \"f159dcf8-4505-42e0-86ce-22e768bdf8a6\" (UID: \"f159dcf8-4505-42e0-86ce-22e768bdf8a6\") "
Jul 02 14:33:41.700748 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:41.700699    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f159dcf8-4505-42e0-86ce-22e768bdf8a6-proxy-tls\") pod \"f159dcf8-4505-42e0-86ce-22e768bdf8a6\" (UID: \"f159dcf8-4505-42e0-86ce-22e768bdf8a6\") "
Jul 02 14:33:41.701055 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:41.701026    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f159dcf8-4505-42e0-86ce-22e768bdf8a6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f159dcf8-4505-42e0-86ce-22e768bdf8a6" (UID: "f159dcf8-4505-42e0-86ce-22e768bdf8a6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:33:41.702629 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:41.702609    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f159dcf8-4505-42e0-86ce-22e768bdf8a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f159dcf8-4505-42e0-86ce-22e768bdf8a6" (UID: "f159dcf8-4505-42e0-86ce-22e768bdf8a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:33:41.801476 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:41.801443    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f159dcf8-4505-42e0-86ce-22e768bdf8a6-openshift-service-ca-bundle\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:33:41.801476 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:41.801471    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f159dcf8-4505-42e0-86ce-22e768bdf8a6-proxy-tls\") on node \"ip-10-0-138-214.ec2.internal\" DevicePath \"\""
Jul 02 14:33:42.371076 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:42.371041    2574 generic.go:356] "Generic (PLEG): container finished" podID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" containerID="bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6" exitCode=0
Jul 02 14:33:42.371272 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:42.371084    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"
Jul 02 14:33:42.371272 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:42.371127    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" event={"ID":"f159dcf8-4505-42e0-86ce-22e768bdf8a6","Type":"ContainerDied","Data":"bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6"}
Jul 02 14:33:42.371272 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:42.371198    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp" event={"ID":"f159dcf8-4505-42e0-86ce-22e768bdf8a6","Type":"ContainerDied","Data":"9bcb1891aa503fd34c10fbb92471b4821e72d37ff9acba0a985fa8a20e3d38b3"}
Jul 02 14:33:42.371272 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:42.371227    2574 scope.go:117] "RemoveContainer" containerID="bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6"
Jul 02 14:33:42.379501 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:42.379486    2574 scope.go:117] "RemoveContainer" containerID="bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6"
Jul 02 14:33:42.379754 ip-10-0-138-214 kubenswrapper[2574]: E0702 14:33:42.379736    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6\": container with ID starting with bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6 not found: ID does not exist" containerID="bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6"
Jul 02 14:33:42.379802 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:42.379762    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6"} err="failed to get container status \"bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6\": rpc error: code = NotFound desc = could not find container \"bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6\": container with ID starting with bfbd39b1099491ae8550f16c07354c4b9144e9e27a8018c7c6c07cd0aa5526d6 not found: ID does not exist"
Jul 02 14:33:42.392576 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:42.392546    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"]
Jul 02 14:33:42.396672 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:42.396653    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8d2ee-545b848957-x45gp"]
Jul 02 14:33:43.351918 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:43.351888    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" path="/var/lib/kubelet/pods/f159dcf8-4505-42e0-86ce-22e768bdf8a6/volumes"
Jul 02 14:33:47.356364 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:47.356297    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h9rfz_77c36603-0176-42fd-a5db-790d4cb28509/global-pull-secret-syncer/0.log"
Jul 02 14:33:47.469717 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:47.469687    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rthpc_11a5eb9e-bf6c-4acc-8566-3047e682e934/konnectivity-agent/0.log"
Jul 02 14:33:47.541546 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:47.541517    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-214.ec2.internal_d47d563e7b550e551dc66f6bd2f8408e/haproxy/0.log"
Jul 02 14:33:51.895143 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:51.895117    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tjnxc_dbbc86cc-3ebd-46fa-826e-04eb377d5643/node-exporter/0.log"
Jul 02 14:33:51.924520 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:51.924493    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tjnxc_dbbc86cc-3ebd-46fa-826e-04eb377d5643/kube-rbac-proxy/0.log"
Jul 02 14:33:51.953883 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:51.953859    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tjnxc_dbbc86cc-3ebd-46fa-826e-04eb377d5643/init-textfile/0.log"
Jul 02 14:33:52.340492 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:52.340463    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-b8npm_9eca96f3-2b55-47e3-9099-4b3b9683c02e/prometheus-operator/0.log"
Jul 02 14:33:52.379813 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:52.379787    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-b8npm_9eca96f3-2b55-47e3-9099-4b3b9683c02e/kube-rbac-proxy/0.log"
Jul 02 14:33:53.974257 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:53.974224    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-2knkg_ef8fe016-fb99-466d-a800-1603da83e1ff/networking-console-plugin/0.log"
Jul 02 14:33:54.519492 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:54.519458    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/1.log"
Jul 02 14:33:54.524097 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:54.524073    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-4frv4_ed604da1-f64a-4102-8272-8cdf4c804988/console-operator/2.log"
Jul 02 14:33:54.958912 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:54.958886    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7bffccbfdd-brlkm_0b96240a-3651-4d67-ade5-1cd9bb698e85/console/0.log"
Jul 02 14:33:55.870937 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.870898    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"]
Jul 02 14:33:55.871423 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871350    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kube-rbac-proxy"
Jul 02 14:33:55.871423 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871369    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kube-rbac-proxy"
Jul 02 14:33:55.871423 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871394    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" containerName="switch-graph-8d2ee"
Jul 02 14:33:55.871423 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871402    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" containerName="switch-graph-8d2ee"
Jul 02 14:33:55.871423 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871416    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" containerName="splitter-graph-78590"
Jul 02 14:33:55.871423 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871425    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" containerName="splitter-graph-78590"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871435    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kserve-container"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871442    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kserve-container"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871455    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kserve-container"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871464    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kserve-container"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871477    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kube-rbac-proxy"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871486    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kube-rbac-proxy"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871571    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kube-rbac-proxy"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871582    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kube-rbac-proxy"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871591    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f159dcf8-4505-42e0-86ce-22e768bdf8a6" containerName="switch-graph-8d2ee"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871600    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6acd2f91-4605-4cfb-87a4-a3828dd4d960" containerName="splitter-graph-78590"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871610    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f294d0b1-01b6-40de-aef0-c6cdb40552a3" containerName="kserve-container"
Jul 02 14:33:55.871736 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.871621    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a1816a-ea79-42c5-af42-a3f922f1526e" containerName="kserve-container"
Jul 02 14:33:55.876193 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.876171    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:55.878722 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.878698    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p8hbs\"/\"kube-root-ca.crt\""
Jul 02 14:33:55.879780 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.879759    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p8hbs\"/\"openshift-service-ca.crt\""
Jul 02 14:33:55.879883 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.879765    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p8hbs\"/\"default-dockercfg-4f72d\""
Jul 02 14:33:55.883999 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:55.883978    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"]
Jul 02 14:33:56.014463 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.014428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flr4\" (UniqueName: \"kubernetes.io/projected/16da28bb-2f6a-437f-84f9-819cc7d3964e-kube-api-access-8flr4\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.014634 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.014480    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-lib-modules\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.014634 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.014508    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-proc\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.014634 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.014570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-podres\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.014746 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.014654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-sys\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.115383 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.115349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-lib-modules\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.115383 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.115379    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-proc\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.115612 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.115399    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-podres\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.115612 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.115428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-sys\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.115612 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.115480    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-proc\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.115612 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.115484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flr4\" (UniqueName: \"kubernetes.io/projected/16da28bb-2f6a-437f-84f9-819cc7d3964e-kube-api-access-8flr4\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.115612 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.115524    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-lib-modules\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.115612 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.115529    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-podres\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.115612 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.115575    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16da28bb-2f6a-437f-84f9-819cc7d3964e-sys\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.124654 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.124598    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flr4\" (UniqueName: \"kubernetes.io/projected/16da28bb-2f6a-437f-84f9-819cc7d3964e-kube-api-access-8flr4\") pod \"perf-node-gather-daemonset-f5qtm\" (UID: \"16da28bb-2f6a-437f-84f9-819cc7d3964e\") " pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.187126 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.187104    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:56.317019 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.316988    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"]
Jul 02 14:33:56.319327 ip-10-0-138-214 kubenswrapper[2574]: W0702 14:33:56.319295    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod16da28bb_2f6a_437f_84f9_819cc7d3964e.slice/crio-80a9c86b417860050fad206b08acec71342813975ce0b713b9001895dbbd5e43 WatchSource:0}: Error finding container 80a9c86b417860050fad206b08acec71342813975ce0b713b9001895dbbd5e43: Status 404 returned error can't find the container with id 80a9c86b417860050fad206b08acec71342813975ce0b713b9001895dbbd5e43
Jul 02 14:33:56.431321 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.431291    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm" event={"ID":"16da28bb-2f6a-437f-84f9-819cc7d3964e","Type":"ContainerStarted","Data":"80a9c86b417860050fad206b08acec71342813975ce0b713b9001895dbbd5e43"}
Jul 02 14:33:56.581512 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.581485    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9fqv7_17913ead-b221-41db-b8e9-8d3d95c929a6/dns/0.log"
Jul 02 14:33:56.620987 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.620968    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9fqv7_17913ead-b221-41db-b8e9-8d3d95c929a6/kube-rbac-proxy/0.log"
Jul 02 14:33:56.878090 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:56.878010    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sdckb_fac51af3-0564-4b8c-946d-7ce5d5bbdf7e/dns-node-resolver/0.log"
Jul 02 14:33:57.436806 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:57.436772    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm" event={"ID":"16da28bb-2f6a-437f-84f9-819cc7d3964e","Type":"ContainerStarted","Data":"98c91de3f206928f079c901a5081397268310dfddd29ba072d453a1e67334527"}
Jul 02 14:33:57.437003 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:57.436856    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:33:57.456866 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:57.456838    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59f6b9978d-hmh9t_7a358c67-3896-4451-b806-e0a2fdd40ad1/registry/0.log"
Jul 02 14:33:57.457022 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:57.456921    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm" podStartSLOduration=2.456906444 podStartE2EDuration="2.456906444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:33:57.455952689 +0000 UTC m=+2974.651576195" watchObservedRunningTime="2026-07-02 14:33:57.456906444 +0000 UTC m=+2974.652529984"
Jul 02 14:33:57.499315 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:57.499289    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d4gsb_fcd1955d-c9c4-4724-ab2c-6d3fec6d9b5c/node-ca/0.log"
Jul 02 14:33:58.856969 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:58.856930    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2knhz_2fbf0c5d-ae9e-4d73-a309-eb3734f78b36/serve-healthcheck-canary/0.log"
Jul 02 14:33:59.292282 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:59.292254    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-s6bjn_b236de76-d2f6-469a-8bed-f59597794af0/insights-operator/0.log"
Jul 02 14:33:59.292647 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:59.292631    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-s6bjn_b236de76-d2f6-469a-8bed-f59597794af0/insights-operator/1.log"
Jul 02 14:33:59.396546 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:59.396521    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n8mlp_7e76678b-b568-49d1-8c10-78d92083fbd9/kube-rbac-proxy/0.log"
Jul 02 14:33:59.420876 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:59.420853    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n8mlp_7e76678b-b568-49d1-8c10-78d92083fbd9/exporter/0.log"
Jul 02 14:33:59.443671 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:33:59.443642    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n8mlp_7e76678b-b568-49d1-8c10-78d92083fbd9/extractor/0.log"
Jul 02 14:34:01.614178 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:01.614129    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7c747fbc6-wfl57_87e3e28f-d41a-49f1-940f-f01fc260cc30/manager/0.log"
Jul 02 14:34:01.664964 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:01.664923    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-m9fs4_534e8181-e835-449f-ad55-0ff46de23e4f/server/0.log"
Jul 02 14:34:01.845262 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:01.845234    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-zslbv_f9ac7535-9962-43af-9a3d-e79a0ff67849/seaweedfs/0.log"
Jul 02 14:34:03.451496 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:03.451470    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p8hbs/perf-node-gather-daemonset-f5qtm"
Jul 02 14:34:06.090192 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:06.090165    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-nmstt_2548bdc9-0467-4364-b2ea-44b75e5ca948/migrator/0.log"
Jul 02 14:34:06.110440 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:06.110412    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-nmstt_2548bdc9-0467-4364-b2ea-44b75e5ca948/graceful-termination/0.log"
Jul 02 14:34:07.765454 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:07.765420    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvh8f_be6b5fc4-8ec4-4bf6-8386-9918c653640a/kube-multus-additional-cni-plugins/0.log"
Jul 02 14:34:07.787320 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:07.787301    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvh8f_be6b5fc4-8ec4-4bf6-8386-9918c653640a/egress-router-binary-copy/0.log"
Jul 02 14:34:07.808009 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:07.807987    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvh8f_be6b5fc4-8ec4-4bf6-8386-9918c653640a/cni-plugins/0.log"
Jul 02 14:34:07.829186 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:07.829161    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvh8f_be6b5fc4-8ec4-4bf6-8386-9918c653640a/bond-cni-plugin/0.log"
Jul 02 14:34:07.850307 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:07.850284    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvh8f_be6b5fc4-8ec4-4bf6-8386-9918c653640a/routeoverride-cni/0.log"
Jul 02 14:34:07.870116 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:07.870098    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvh8f_be6b5fc4-8ec4-4bf6-8386-9918c653640a/whereabouts-cni-bincopy/0.log"
Jul 02 14:34:07.890259 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:07.890227    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wvh8f_be6b5fc4-8ec4-4bf6-8386-9918c653640a/whereabouts-cni/0.log"
Jul 02 14:34:07.922094 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:07.922067    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nqrn8_8319cd50-4436-4a2c-9a5d-0c8c740c0fbc/kube-multus/0.log"
Jul 02 14:34:07.988180 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:07.988158    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6vhv6_fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de/network-metrics-daemon/0.log"
Jul 02 14:34:08.007586 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:08.007563    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6vhv6_fb870114-0df5-4eb1-bb2a-3ac9e7e2e5de/kube-rbac-proxy/0.log"
Jul 02 14:34:09.023154 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:09.023109    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-controller/0.log"
Jul 02 14:34:09.041514 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:09.041488    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/1.log"
Jul 02 14:34:09.041658 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:09.041526    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovn-acl-logging/0.log"
Jul 02 14:34:09.060616 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:09.060591    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/kube-rbac-proxy-node/0.log"
Jul 02 14:34:09.082970 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:09.082951    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 14:34:09.102995 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:09.102976    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/northd/0.log"
Jul 02 14:34:09.127028 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:09.126999    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/nbdb/0.log"
Jul 02 14:34:09.151847 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:09.151819    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/sbdb/0.log"
Jul 02 14:34:09.219116 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:09.219086    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lmvr2_9310de27-67d9-48e4-9628-ac3302f1b57d/ovnkube-controller/0.log"
Jul 02 14:34:10.494354 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:10.494323    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-tlx9j_db50e6b2-f690-4c96-a2f6-1843a40e95ec/network-check-target-container/0.log"
Jul 02 14:34:11.412346 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:11.412313    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-tmnd6_434a4424-1ea6-4226-8665-6178bcdc10a2/iptables-alerter/0.log"
Jul 02 14:34:12.037227 ip-10-0-138-214 kubenswrapper[2574]: I0702 14:34:12.037199    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-lw6s2_a13e4959-f5b7-4ad9-a9c1-462cb376231b/tuned/0.log"