Jul 09 18:02:14.020675 ip-10-0-140-234 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 09 18:02:14.020687 ip-10-0-140-234 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 09 18:02:14.020694 ip-10-0-140-234 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 09 18:02:14.020924 ip-10-0-140-234 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 09 18:02:25.318226 ip-10-0-140-234 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 09 18:02:25.318241 ip-10-0-140-234 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ac8ed8723bcd404d951590b885a597f8 --
Jul 09 18:04:05.528179 ip-10-0-140-234 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.988630    2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995122    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995139    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995142    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995146    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995149    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995152    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995156    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:05.995434 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995160    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995163    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995166    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995169    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995172    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995174    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995177    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995179    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995182    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995185    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995188    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995191    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995194    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995196    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995199    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995205    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995209    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995212    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995215    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:05.998139 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995217    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995220    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995223    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995225    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995228    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995231    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995234    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995237    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995240    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995243    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995246    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995249    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995251    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995254    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995256    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995260    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995265    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995268    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995271    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:06.011622 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995273    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995276    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995279    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995281    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995285    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995288    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995291    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995294    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995296    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995300    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995303    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995306    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995308    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995311    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995314    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995316    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995319    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995323    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995326    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995329    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:06.014111 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995333    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995336    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995340    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995343    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995345    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995348    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995351    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995356    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995359    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995361    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995364    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995367    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995371    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995373    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995376    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995379    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995381    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995384    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995386    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995389    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:06.050401 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995392    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995395    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995398    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995400    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995403    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995405    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995408    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995411    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995414    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995417    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995419    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995422    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995425    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.995427    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996187    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996196    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996199    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996202    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996205    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996209    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:06.056045 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996211    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996214    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996217    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996219    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996222    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996225    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996228    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996230    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996234    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996238    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996241    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996244    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996246    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996249    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996252    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996255    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996258    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996260    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996262    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996265    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:06.059617 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996268    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996270    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996273    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996276    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996278    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996281    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996283    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996286    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996288    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996290    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996293    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996296    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996299    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996301    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996304    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996307    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996309    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996312    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996314    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996317    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:06.061406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996319    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996322    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996324    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996327    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996329    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996332    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996334    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996338    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996340    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996343    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996345    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996347    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996350    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996353    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996357    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996360    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996363    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996365    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996368    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996370    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:06.062996 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996373    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996375    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996378    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996380    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996383    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996386    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996388    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996390    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996393    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996396    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996400    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996404    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996407    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996410    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996414    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996417    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996419    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996422    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996424    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:06.064615 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996427    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996429    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996432    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996434    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996437    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996439    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996441    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996444    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996446    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996449    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996451    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996454    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996456    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996458    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996821    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996826    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996829    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996832    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996835    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996838    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:06.071108 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996841    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996843    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996847    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996851    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996854    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996857    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996859    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996863    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996865    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996868    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996870    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996873    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996876    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996878    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996881    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996903    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996908    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996912    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996916    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:06.084671 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996919    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:06.075238 ip-10-0-140-234 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996922    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996924    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996927    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996930    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996932    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996935    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996938    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996941    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996943    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996946    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996949    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996951    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996954    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996957    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996960    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996962    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996965    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996969    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996971    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:06.109431 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996974    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996977    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996979    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996982    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996985    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996987    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996990    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996992    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996995    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.996998    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997000    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997003    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997005    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997007    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997010    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997012    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997015    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997018    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997022    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997024    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:06.124275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997027    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997029    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997032    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997034    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997037    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997040    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997042    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997045    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997048    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997052    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997055    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997059    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997062    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997065    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997068    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997071    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997073    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997076    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997078    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:06.319029 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997081    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997083    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997085    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997088    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997090    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997094    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997096    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997098    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997101    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997103    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997106    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997109    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997112    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997114    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997117    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997186    2568 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997193    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997199    2568 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997203    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997207    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997211    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:04:06.333340 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997215    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997219    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997222    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997225    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997229    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997233    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997236    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997239    2568 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997242    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997245    2568 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997247    2568 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997251    2568 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997254    2568 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997257    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997260    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997263    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997266    2568 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997270    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997273    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997276    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997282    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997285    2568 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997288    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997291    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:04:06.334774 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997294    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997297    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997301    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997304    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997307    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997310    2568 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997312    2568 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997315    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997320    2568 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997323    2568 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997326    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997329    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997331    2568 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997335    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997338    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997341    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997344    2568 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997347    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997350    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997353    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997356    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997358    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997361    2568 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997364    2568 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997367    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:04:06.335642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997370    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997373    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997376    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997381    2568 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997384    2568 flags.go:64] FLAG: --help="false"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997388    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997391    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997394    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997397    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997400    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997403    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997406    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997409    2568 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997411    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997414    2568 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997417    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997420    2568 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997423    2568 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997426    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997428    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997431    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997434    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997437    2568 flags.go:64] FLAG: --lock-file=""
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997440    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:04:06.337261 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997443    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997446    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997451    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997453    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997456    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997459    2568 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997462    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997465    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997467    2568 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997470    2568 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997474    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997479    2568 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997483    2568 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997488    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997491    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997493    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997496    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997499    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997502    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997505    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997513    2568 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997517    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997520    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:04:06.338029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997523    2568 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997526    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997531    2568 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997534    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997537    2568 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997540    2568 flags.go:64] FLAG: --port="10250"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997543    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997550    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0578fb1618598b999"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997553    2568 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997556    2568 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997559    2568 flags.go:64] FLAG: --register-node="true"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997562    2568 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997566    2568 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997569    2568 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997571    2568 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997574    2568 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997578    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997580    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997583    2568 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997586    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997590    2568 flags.go:64] FLAG: --runonce="false"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997593    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997596    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997601    2568 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997604    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:04:06.338934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997607    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997609    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997612    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997615    2568 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997618    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997621    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997623    2568 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997626    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997629    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997632    2568 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997635    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997640    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997643    2568 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997646    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997649    2568 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997652    2568 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997655    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997658    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997660    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997663    2568 flags.go:64] FLAG: --v="2"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997667    2568 flags.go:64] FLAG: --version="false"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997671    2568 flags.go:64] FLAG: --vmodule=""
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997674    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.997678    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997789    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:06.340624 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997793    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997796    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997800    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997803    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997806    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997810    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997812    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997815    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997817    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997820    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997822    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997825    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997828    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997830    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997833    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997836    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997838    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997841    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997843    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997846    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:06.341580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997848    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997850    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997853    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997856    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997858    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997861    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997863    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997866    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997868    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997871    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997873    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997875    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997878    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997880    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997897    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997900    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997902    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997906    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997908    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:06.342272 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997911    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997914    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997916    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997919    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997921    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997924    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997926    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997929    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997931    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997934    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997937    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997939    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997942    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997944    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997947    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997949    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997952    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997955    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997958    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997960    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:06.343037 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997963    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997966    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997968    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997971    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997974    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997978    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997982    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997985    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997988    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997991    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997994    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997997    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.997999    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998002    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998005    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998007    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998010    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998013    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:06.343640 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998017    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998021    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998024    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998027    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998030    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998032    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998035    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998037    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998040    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998042    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998045    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998047    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998050    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998053    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998055    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998058    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998060    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998063    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998066    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998068    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:06.344406 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:05.998071    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:05.998834    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.006262    2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.006275    2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.006965    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.006975    2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007041    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007046    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007049    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007052    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007055    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007058    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007061    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007065    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:06.345057 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007069    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007072    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007075    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007077    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007080    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007082    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007085    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007087    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007090    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007093    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007101    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007104    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007107    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007110    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007112    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007115    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007117    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007120    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007122    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007127    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:06.345724 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007129    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007132    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007134    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007138    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007141    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007143    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007146    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007148    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007151    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007153    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007156    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007158    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007161    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007163    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007166    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007169    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007171    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007174    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007176    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007178    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:06.346347 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007184    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007187    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007190    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007202    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007206    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007209    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007212    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007214    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007218    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007221    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007224    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007228    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007231    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007233    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007236    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007240    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007243    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007245    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007248    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:06.347092 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007250    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007253    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007256    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007258    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007260    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007263    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007274    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007277    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007279    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007281    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007284    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007287    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007289    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007292    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007294    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007296    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007299    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007307    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007310    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:06.502828 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007313    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007315    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007318    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007320    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007322    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007326    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007329    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007331    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007333    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007338    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007340    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007343    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007345    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.007350    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007525    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:06.504276 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007530    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007533    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007535    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007538    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007541    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007544    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007547    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007549    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007552    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007554    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007557    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007559    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007562    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007564    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007567    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007569    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007581    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007584    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007586    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:06.504738 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007589    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007591    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007594    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007600    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007603    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007606    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007609    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007613    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007618    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007621    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007624    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007626    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007629    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007632    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007634    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007637    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007639    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007642    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007644    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:06.505510 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007647    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007649    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007652    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007654    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007656    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007659    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007662    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007664    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007667    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007669    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007672    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007679    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007682    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007685    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007687    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007691    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007695    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007699    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007702    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007705    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:06.506073 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007707    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007712    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007715    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007718    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007720    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007723    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007726    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007728    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007731    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007734    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007736    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007739    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007741    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007744    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007746    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007749    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007751    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007754    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007757    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007759    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:06.506869 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007762    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007764    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007767    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007769    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007777    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007780    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007783    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007785    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007790    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007793    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007795    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007798    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007800    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007805    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007808    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007811    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007814    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007816    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007819    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:06.507439 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:06.007821    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.007827    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.007956    2568 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.010216    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.011523    2568 server.go:1019] "Starting client certificate rotation"
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.011656    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.011724    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.014195    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.019408    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.037296    2568 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.039313    2568 log.go:25] "Validated CRI v1 image API"
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.040103    2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:04:06.508157 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.044195    2568 fs.go:135] Filesystem UUIDs: map[29a8090c-91a4-4798-ac6f-4db346e9d893:/dev/nvme0n1p3 4f896612-087c-494c-960d-0b03d6b7762b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 09 18:04:06.508522 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.044222    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:04:06.508522 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.049445    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.050353    2568 manager.go:216] Machine: {Timestamp:2026-07-09 18:04:06.048772428 +0000 UTC m=+0.382636498 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100092 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cf06069bc8b045b6a2d9a69d4a0b3 SystemUUID:ec2cf060-69bc-8b04-5b6a-2d9a69d4a0b3 BootID:ac8ed872-3bcd-404d-9515-90b885a597f8 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f9:18:f8:de:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f9:18:f8:de:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:de:fe:66:ad:9a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.050461    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.050544    2568 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.054019    2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.054044    2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-234.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.055386    2568 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.055398    2568 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.055408    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.056122    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.057092    2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.060718    2568 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.060845    2568 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.060874    2568 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.060907    2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.060921    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.060933    2568 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.060944    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:04:06.508855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.061316    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.061336    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.063789    2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.065363    2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.065399    2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066160    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066180    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066191    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066223    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066240    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066247    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066259    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066267    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066276    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066283    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066310    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.066317    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.067465    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.067475    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.073277    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7krnw"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.074159    2568 server.go:1268] "Started kubelet"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.074310    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.074332    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.074365    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.074925    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-234.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.075863    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.075915    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.075920    2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.076332    2568 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.081078    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7krnw"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.082470    2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.083350    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-234.ec2.internal.18c0b129ed7168c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-234.ec2.internal,UID:ip-10-0-140-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-234.ec2.internal,},FirstTimestamp:2026-07-09 18:04:06.074124488 +0000 UTC m=+0.407988560,LastTimestamp:2026-07-09 18:04:06.074124488 +0000 UTC m=+0.407988560,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-234.ec2.internal,}"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.084950    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.084962    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.085067    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.085561    2568 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.085576    2568 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.085705    2568 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.085713    2568 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.085712    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:04:06.647442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.086437    2568 factory.go:55] Registering systemd factory
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.086452    2568 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.086682    2568 factory.go:153] Registering CRI-O factory
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.086695    2568 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.086747    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.086767    2568 factory.go:103] Registering Raw factory
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.086776    2568 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.087184    2568 manager.go:349] Starting recovery of all containers
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.098248    2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.100196    2568 manager.go:354] Recovery completed
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.100854    2568 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-234.ec2.internal\" not found" node="ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.106516    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.108839    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.108866    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.108879    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.109544    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.109559    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.109558    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.109665    2568 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.110281    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.110306    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.110316    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.111001    2568 policy_none.go:49] "None policy: Start"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.111018    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.111032    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.112362    2568 policy_none.go:47] "Start"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.164374    2568 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.164498    2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.164508    2568 server.go:87] "Starting device plugin registration server"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.164697    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.164706    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.164817    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.164920    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.164929    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.165392    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.165437    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.219609    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.220818    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.220855    2568 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.220881    2568 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.220936    2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.223072    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.264780    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.265682    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.265711    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.265722    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.265745    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.273848    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.273867    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-234.ec2.internal\": node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:06.648639 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.291663    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.321923    2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-234.ec2.internal"]
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.321991    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.323702    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.323730    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.323740    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.325232    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.325335    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.325360    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.325905    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.325928    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.325908    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.325949    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.325960    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.325972    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.327529    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.327556    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.328205    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.328232    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.328246    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.360458    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-234.ec2.internal\" not found" node="ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.368272    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-234.ec2.internal\" not found" node="ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.387284    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46a34aa9fbfdc20a772f622b306ebd77-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal\" (UID: \"46a34aa9fbfdc20a772f622b306ebd77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.387304    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46a34aa9fbfdc20a772f622b306ebd77-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal\" (UID: \"46a34aa9fbfdc20a772f622b306ebd77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.387324    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27720bce3ffdbdb8ce1870f3c11220d9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-234.ec2.internal\" (UID: \"27720bce3ffdbdb8ce1870f3c11220d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.392373    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.487593    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46a34aa9fbfdc20a772f622b306ebd77-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal\" (UID: \"46a34aa9fbfdc20a772f622b306ebd77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.487619    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46a34aa9fbfdc20a772f622b306ebd77-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal\" (UID: \"46a34aa9fbfdc20a772f622b306ebd77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.650287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.487637    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27720bce3ffdbdb8ce1870f3c11220d9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-234.ec2.internal\" (UID: \"27720bce3ffdbdb8ce1870f3c11220d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.651119 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.487677    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27720bce3ffdbdb8ce1870f3c11220d9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-234.ec2.internal\" (UID: \"27720bce3ffdbdb8ce1870f3c11220d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.651119 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.487697    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/46a34aa9fbfdc20a772f622b306ebd77-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal\" (UID: \"46a34aa9fbfdc20a772f622b306ebd77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.651119 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.487708    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46a34aa9fbfdc20a772f622b306ebd77-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal\" (UID: \"46a34aa9fbfdc20a772f622b306ebd77\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.651119 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.492684    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:06.651119 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.592860    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:06.774026 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.663050    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.774026 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:06.671002    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:06.774026 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.692981    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:06.937793 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.793414    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:06.937793 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.894013    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:06.994180 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:06.994136    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:07.011598 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.011574    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:04:07.012141 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.011739    2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:07.012141 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.011771    2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:07.088849 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.088807    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:59:06 +0000 UTC" deadline="2027-12-26 10:33:48.46391561 +0000 UTC"
Jul 09 18:04:07.088849 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.088838    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12832h29m41.375080997s"
Jul 09 18:04:07.095335 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:07.095314    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:07.196032 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:07.196005    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:07.296160 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:07.296125    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-234.ec2.internal\" not found"
Jul 09 18:04:07.314676 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:07.314637    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27720bce3ffdbdb8ce1870f3c11220d9.slice/crio-b5c637a3432ee7b55fc331991b32e43c4f3c5f100b85b43a141e18e084ea4e85 WatchSource:0}: Error finding container b5c637a3432ee7b55fc331991b32e43c4f3c5f100b85b43a141e18e084ea4e85: Status 404 returned error can't find the container with id b5c637a3432ee7b55fc331991b32e43c4f3c5f100b85b43a141e18e084ea4e85
Jul 09 18:04:07.321522 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.321507    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:04:07.335974 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.335942    2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:07.345238 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.345220    2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:07.387842 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.387812    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:07.400657 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.400638    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:07.402014 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.401999    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-234.ec2.internal"
Jul 09 18:04:07.413014 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.412994    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:07.423384 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:07.423361    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a34aa9fbfdc20a772f622b306ebd77.slice/crio-c925d07184e698d0b8a16331a5694e30bdb505753157635ed49fdd0ad6770218 WatchSource:0}: Error finding container c925d07184e698d0b8a16331a5694e30bdb505753157635ed49fdd0ad6770218: Status 404 returned error can't find the container with id c925d07184e698d0b8a16331a5694e30bdb505753157635ed49fdd0ad6770218
Jul 09 18:04:07.878424 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.878392    2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:07.882029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:07.882008    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:08.063716 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.063685    2568 apiserver.go:52] "Watching apiserver"
Jul 09 18:04:08.073082 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.073058    2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:04:08.074150 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.074127    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-wvzlz","openshift-multus/multus-additional-cni-plugins-npq7x","openshift-multus/multus-wvqdj","openshift-multus/network-metrics-daemon-f84z2","openshift-network-diagnostics/network-check-target-bth88","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb","openshift-dns/node-resolver-kg2kp","openshift-image-registry/node-ca-jlzk7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal","openshift-network-operator/iptables-alerter-bkn7f","openshift-ovn-kubernetes/ovnkube-node-bstvh","kube-system/global-pull-secret-syncer-pqd8h","kube-system/konnectivity-agent-xm9lj","kube-system/kube-apiserver-proxy-ip-10-0-140-234.ec2.internal"]
Jul 09 18:04:08.077000 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.076973    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bkn7f"
Jul 09 18:04:08.079246 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.079226    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:08.080271 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.080253    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:08.080368 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.080293    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkwjl\""
Jul 09 18:04:08.080368 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.080355    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:04:08.080597 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.080581    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:08.081492 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.081476    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.082008 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.081990    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:04:08.082448 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.082433    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8dndp\""
Jul 09 18:04:08.082571 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.082551    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:04:08.083699 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.083681    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:08.083790 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.083743    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:08.084595 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.084571    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:04:08.084740 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.084625    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:04:08.084882 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.084865    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:04:08.084947 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.084932    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-flpst\""
Jul 09 18:04:08.085017 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.084990    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:04:08.085694 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.085676    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:08.085977 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.085957    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:08.086053 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.086022    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:08.088347 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.088329    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.090706 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.090686    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jlzk7"
Jul 09 18:04:08.091083 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.091060    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:04:08.091531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.091427    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j49ws\""
Jul 09 18:04:08.091531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.091429    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:04:08.091646 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.091634    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:04:08.093139 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.093119    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.095907 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.095888    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:04:08.096228 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.096207    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:04:08.096712 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.096355    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7x97p\""
Jul 09 18:04:08.096712 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.096361    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:04:08.096712 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.096581    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:08.096712 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.096619    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-system-cni-dir\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.096712 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.096665    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-multus-cni-dir\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.096986 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.096726    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-var-lib-cni-multus\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.096986 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.096847    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-var-lib-kubelet\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097257 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097236    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/148bb254-e170-4306-9b3f-10811338d4b2-multus-daemon-config\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097361 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097270    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-etc-kubernetes\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097361 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097298    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfl2l\" (UniqueName: \"kubernetes.io/projected/148bb254-e170-4306-9b3f-10811338d4b2-kube-api-access-gfl2l\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097465 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097357    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvpw4\" (UniqueName: \"kubernetes.io/projected/c4b61885-5a4f-4355-ab6c-f95538b7992b-kube-api-access-mvpw4\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:08.097465 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097419    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-run-k8s-cni-cncf-io\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097541 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097461    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc692\" (UniqueName: \"kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692\") pod \"network-check-target-bth88\" (UID: \"113c3ee7-a95e-42b6-87d8-68149f3382c8\") " pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:08.097541 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097491    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfx7n\" (UniqueName: \"kubernetes.io/projected/ea4d5a34-f150-4a58-ad65-60b4c5f57295-kube-api-access-zfx7n\") pod \"iptables-alerter-bkn7f\" (UID: \"ea4d5a34-f150-4a58-ad65-60b4c5f57295\") " pod="openshift-network-operator/iptables-alerter-bkn7f"
Jul 09 18:04:08.097541 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097518    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/087b34f5-8d0f-4612-9f19-a87dd8fbf564-konnectivity-ca\") pod \"konnectivity-agent-xm9lj\" (UID: \"087b34f5-8d0f-4612-9f19-a87dd8fbf564\") " pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:08.097649 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097590    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-cnibin\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097649 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097631    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/148bb254-e170-4306-9b3f-10811338d4b2-cni-binary-copy\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097731 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097652    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea4d5a34-f150-4a58-ad65-60b4c5f57295-iptables-alerter-script\") pod \"iptables-alerter-bkn7f\" (UID: \"ea4d5a34-f150-4a58-ad65-60b4c5f57295\") " pod="openshift-network-operator/iptables-alerter-bkn7f"
Jul 09 18:04:08.097731 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097669    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea4d5a34-f150-4a58-ad65-60b4c5f57295-host-slash\") pod \"iptables-alerter-bkn7f\" (UID: \"ea4d5a34-f150-4a58-ad65-60b4c5f57295\") " pod="openshift-network-operator/iptables-alerter-bkn7f"
Jul 09 18:04:08.097731 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097689    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/087b34f5-8d0f-4612-9f19-a87dd8fbf564-agent-certs\") pod \"konnectivity-agent-xm9lj\" (UID: \"087b34f5-8d0f-4612-9f19-a87dd8fbf564\") " pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:08.097731 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097712    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-os-release\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097887 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097739    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-multus-socket-dir-parent\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097887 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097754    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-run-netns\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097887 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097770    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-var-lib-cni-bin\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097887 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097786    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-hostroot\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.097887 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097840    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-multus-conf-dir\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.098127 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.097909    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-run-multus-certs\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.098182 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.098121    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:04:08.098182 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.098142    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.098182 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.098167    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:04:08.098371 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.098347    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:04:08.098430 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.098409    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:04:08.098486 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.098439    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:04:08.098534 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.098408    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:04:08.098582 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.098561    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7trbd\""
Jul 09 18:04:08.100881 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.100859    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.101819 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.101801    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:04:08.102328 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.102309    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:04:08.102864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.102847    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fl585\""
Jul 09 18:04:08.103375 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.103348    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kg2kp"
Jul 09 18:04:08.103817 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.103794    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:08.104116 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.104100    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h6pfk\""
Jul 09 18:04:08.104217 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.104207    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:08.106004 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.105957    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:08.106098 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.106050    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:08.107222 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.107197    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:08.107607 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.107549    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:04:08.107806 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.107787    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:04:08.107993 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.107794    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-58jcx\""
Jul 09 18:04:08.132242 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.132216    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j8c6m"
Jul 09 18:04:08.140259 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.140234    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j8c6m"
Jul 09 18:04:08.188216 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.188143    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:04:08.199102 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199068    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-lib-modules\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.199276 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199111    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-var-lib-kubelet\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.199276 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199141    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:08.199276 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199172    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-multus-socket-dir-parent\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.199276 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199198    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-etc-selinux\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.199276 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199223    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-node-log\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.199276 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199246    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-log-socket\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.199276 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199269    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-ovnkube-script-lib\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199294    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199317    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-sysctl-conf\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199342    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-multus-cni-dir\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199367    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-var-lib-kubelet\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199392    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199415    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-run-netns\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199424    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-multus-socket-dir-parent\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199437    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-run-ovn-kubernetes\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199485    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-var-lib-kubelet\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199489    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-run-k8s-cni-cncf-io\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199521    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-systemd-units\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199541    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-multus-cni-dir\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199548    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln7rh\" (UniqueName: \"kubernetes.io/projected/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-kube-api-access-ln7rh\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199556    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-run-k8s-cni-cncf-io\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.199578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199576    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-cni-binary-copy\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199615    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-tuned\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199653    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-device-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199688    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-system-cni-dir\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199712    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-etc-openvswitch\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199738    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-var-lib-cni-bin\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199758    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-var-lib-openvswitch\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199773    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-sysctl-d\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199789    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cda30b50-0c11-4e28-8ea9-a954ae1b8089-tmp\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199791    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-var-lib-cni-bin\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199804    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10-hosts-file\") pod \"node-resolver-kg2kp\" (UID: \"24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10\") " pod="openshift-dns/node-resolver-kg2kp"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199826    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-sys-fs\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199853    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-os-release\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199885    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10-tmp-dir\") pod \"node-resolver-kg2kp\" (UID: \"24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10\") " pod="openshift-dns/node-resolver-kg2kp"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199930    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2wh\" (UniqueName: \"kubernetes.io/projected/24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10-kube-api-access-pk2wh\") pod \"node-resolver-kg2kp\" (UID: \"24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10\") " pod="openshift-dns/node-resolver-kg2kp"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199960    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc692\" (UniqueName: \"kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692\") pod \"network-check-target-bth88\" (UID: \"113c3ee7-a95e-42b6-87d8-68149f3382c8\") " pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.199992    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-slash\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.200236 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200026    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/148bb254-e170-4306-9b3f-10811338d4b2-cni-binary-copy\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200050    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-kubelet\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200074    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-cnibin\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200098    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-sysconfig\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200120    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-run\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200138    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-sys\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200156    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea4d5a34-f150-4a58-ad65-60b4c5f57295-iptables-alerter-script\") pod \"iptables-alerter-bkn7f\" (UID: \"ea4d5a34-f150-4a58-ad65-60b4c5f57295\") " pod="openshift-network-operator/iptables-alerter-bkn7f"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200176    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/087b34f5-8d0f-4612-9f19-a87dd8fbf564-agent-certs\") pod \"konnectivity-agent-xm9lj\" (UID: \"087b34f5-8d0f-4612-9f19-a87dd8fbf564\") " pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200192    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-os-release\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200209    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200224    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-env-overrides\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200241    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200256    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd84a8bd-6c02-477b-8e99-6b78466bea4a-dbus\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200273    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-ovnkube-config\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200316    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-var-lib-cni-multus\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200696    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:04:08.201044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.200971    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/148bb254-e170-4306-9b3f-10811338d4b2-cni-binary-copy\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.201083    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201123    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-os-release\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.201197    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs podName:c4b61885-5a4f-4355-ab6c-f95538b7992b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:08.701173867 +0000 UTC m=+3.035037934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs") pod "network-metrics-daemon-f84z2" (UID: "c4b61885-5a4f-4355-ab6c-f95538b7992b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201220    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfl2l\" (UniqueName: \"kubernetes.io/projected/148bb254-e170-4306-9b3f-10811338d4b2-kube-api-access-gfl2l\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201223    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-var-lib-cni-multus\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201260    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfx7n\" (UniqueName: \"kubernetes.io/projected/ea4d5a34-f150-4a58-ad65-60b4c5f57295-kube-api-access-zfx7n\") pod \"iptables-alerter-bkn7f\" (UID: \"ea4d5a34-f150-4a58-ad65-60b4c5f57295\") " pod="openshift-network-operator/iptables-alerter-bkn7f"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201294    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-cni-bin\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201317    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-modprobe-d\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201350    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-socket-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201372    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-registration-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201394    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvm6m\" (UniqueName: \"kubernetes.io/projected/b3bd61c5-6cd4-434e-8b3d-487cbacf897f-kube-api-access-bvm6m\") pod \"node-ca-jlzk7\" (UID: \"b3bd61c5-6cd4-434e-8b3d-487cbacf897f\") " pod="openshift-image-registry/node-ca-jlzk7"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201416    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-run-systemd\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201436    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-cni-netd\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201457    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-ovn-node-metrics-cert\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201476    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-host\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201498    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9lb4\" (UniqueName: \"kubernetes.io/projected/cda30b50-0c11-4e28-8ea9-a954ae1b8089-kube-api-access-t9lb4\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.201864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201523    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-cnibin\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201546    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvpw4\" (UniqueName: \"kubernetes.io/projected/c4b61885-5a4f-4355-ab6c-f95538b7992b-kube-api-access-mvpw4\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201569    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea4d5a34-f150-4a58-ad65-60b4c5f57295-host-slash\") pod \"iptables-alerter-bkn7f\" (UID: \"ea4d5a34-f150-4a58-ad65-60b4c5f57295\") " pod="openshift-network-operator/iptables-alerter-bkn7f"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201590    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-run-netns\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201612    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-hostroot\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201632    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-multus-conf-dir\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201653    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-run-multus-certs\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201675    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201689    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-run-netns\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201696    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-system-cni-dir\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201743    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-system-cni-dir\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201744    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/148bb254-e170-4306-9b3f-10811338d4b2-multus-daemon-config\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201775    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-etc-kubernetes\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201793    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea4d5a34-f150-4a58-ad65-60b4c5f57295-host-slash\") pod \"iptables-alerter-bkn7f\" (UID: \"ea4d5a34-f150-4a58-ad65-60b4c5f57295\") " pod="openshift-network-operator/iptables-alerter-bkn7f"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201810    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3bd61c5-6cd4-434e-8b3d-487cbacf897f-host\") pod \"node-ca-jlzk7\" (UID: \"b3bd61c5-6cd4-434e-8b3d-487cbacf897f\") " pod="openshift-image-registry/node-ca-jlzk7"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201840    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3bd61c5-6cd4-434e-8b3d-487cbacf897f-serviceca\") pod \"node-ca-jlzk7\" (UID: \"b3bd61c5-6cd4-434e-8b3d-487cbacf897f\") " pod="openshift-image-registry/node-ca-jlzk7"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201841    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-cnibin\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.202807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201862    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201885    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-systemd\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201890    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-multus-conf-dir\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201892    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-hostroot\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201847    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-host-run-multus-certs\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201911    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/087b34f5-8d0f-4612-9f19-a87dd8fbf564-konnectivity-ca\") pod \"konnectivity-agent-xm9lj\" (UID: \"087b34f5-8d0f-4612-9f19-a87dd8fbf564\") " pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201928    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/148bb254-e170-4306-9b3f-10811338d4b2-etc-kubernetes\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201936    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-run-ovn\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.201976    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-kubernetes\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.202011    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd84a8bd-6c02-477b-8e99-6b78466bea4a-kubelet-config\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.202044    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlbzv\" (UniqueName: \"kubernetes.io/projected/b7a65ab8-59f7-4487-9329-7352324a3319-kube-api-access-qlbzv\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.202078    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-run-openvswitch\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.202103    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8b8w\" (UniqueName: \"kubernetes.io/projected/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-kube-api-access-t8b8w\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.202496    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/148bb254-e170-4306-9b3f-10811338d4b2-multus-daemon-config\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.202595    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/087b34f5-8d0f-4612-9f19-a87dd8fbf564-konnectivity-ca\") pod \"konnectivity-agent-xm9lj\" (UID: \"087b34f5-8d0f-4612-9f19-a87dd8fbf564\") " pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:08.203665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.202837    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea4d5a34-f150-4a58-ad65-60b4c5f57295-iptables-alerter-script\") pod \"iptables-alerter-bkn7f\" (UID: \"ea4d5a34-f150-4a58-ad65-60b4c5f57295\") " pod="openshift-network-operator/iptables-alerter-bkn7f"
Jul 09 18:04:08.205695 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.205673    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/087b34f5-8d0f-4612-9f19-a87dd8fbf564-agent-certs\") pod \"konnectivity-agent-xm9lj\" (UID: \"087b34f5-8d0f-4612-9f19-a87dd8fbf564\") " pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:08.210526 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.210508    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:08.210628 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.210530    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:08.210628 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.210543    2568 projected.go:196] Error preparing data for projected volume kube-api-access-jc692 for pod openshift-network-diagnostics/network-check-target-bth88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:08.210628 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.210602    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692 podName:113c3ee7-a95e-42b6-87d8-68149f3382c8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:08.710584857 +0000 UTC m=+3.044448936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jc692" (UniqueName: "kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692") pod "network-check-target-bth88" (UID: "113c3ee7-a95e-42b6-87d8-68149f3382c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:08.213622 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.213595    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfx7n\" (UniqueName: \"kubernetes.io/projected/ea4d5a34-f150-4a58-ad65-60b4c5f57295-kube-api-access-zfx7n\") pod \"iptables-alerter-bkn7f\" (UID: \"ea4d5a34-f150-4a58-ad65-60b4c5f57295\") " pod="openshift-network-operator/iptables-alerter-bkn7f"
Jul 09 18:04:08.214098 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.214078    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfl2l\" (UniqueName: \"kubernetes.io/projected/148bb254-e170-4306-9b3f-10811338d4b2-kube-api-access-gfl2l\") pod \"multus-wvqdj\" (UID: \"148bb254-e170-4306-9b3f-10811338d4b2\") " pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.215093 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.215060    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvpw4\" (UniqueName: \"kubernetes.io/projected/c4b61885-5a4f-4355-ab6c-f95538b7992b-kube-api-access-mvpw4\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:08.227894 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.227403    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal" event={"ID":"46a34aa9fbfdc20a772f622b306ebd77","Type":"ContainerStarted","Data":"c925d07184e698d0b8a16331a5694e30bdb505753157635ed49fdd0ad6770218"}
Jul 09 18:04:08.229502 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.229446    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-234.ec2.internal" event={"ID":"27720bce3ffdbdb8ce1870f3c11220d9","Type":"ContainerStarted","Data":"b5c637a3432ee7b55fc331991b32e43c4f3c5f100b85b43a141e18e084ea4e85"}
Jul 09 18:04:08.302799 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.302758    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-kubelet\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.303025 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.302832    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-cnibin\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.303025 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.302860    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-sysconfig\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.303025 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.302883    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-run\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.303025 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.302921    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-sys\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.303025 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303018    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-cnibin\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.303308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303059    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-sysconfig\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.303308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303065    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-kubelet\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.303308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303085    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-env-overrides\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.303308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303070    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-run\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.303308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303129    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-sys\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.303308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303175    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.303308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303205    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd84a8bd-6c02-477b-8e99-6b78466bea4a-dbus\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:08.303308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303231    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-ovnkube-config\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.303308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303263    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-cni-bin\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.303308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303291    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-modprobe-d\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303316    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-socket-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303327    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd84a8bd-6c02-477b-8e99-6b78466bea4a-dbus\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303343    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-registration-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303372    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvm6m\" (UniqueName: \"kubernetes.io/projected/b3bd61c5-6cd4-434e-8b3d-487cbacf897f-kube-api-access-bvm6m\") pod \"node-ca-jlzk7\" (UID: \"b3bd61c5-6cd4-434e-8b3d-487cbacf897f\") " pod="openshift-image-registry/node-ca-jlzk7"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303367    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303377    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-cni-bin\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303490    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-modprobe-d\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303492    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-registration-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303609    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-run-systemd\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303639    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-cni-netd\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303674    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-ovn-node-metrics-cert\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303682    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-socket-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303687    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-run-systemd\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303699    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-host\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303739    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-cni-netd\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303747    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9lb4\" (UniqueName: \"kubernetes.io/projected/cda30b50-0c11-4e28-8ea9-a954ae1b8089-kube-api-access-t9lb4\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.303795 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303711    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-env-overrides\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303784    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-host\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303793    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303859    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3bd61c5-6cd4-434e-8b3d-487cbacf897f-host\") pod \"node-ca-jlzk7\" (UID: \"b3bd61c5-6cd4-434e-8b3d-487cbacf897f\") " pod="openshift-image-registry/node-ca-jlzk7"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303920    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3bd61c5-6cd4-434e-8b3d-487cbacf897f-serviceca\") pod \"node-ca-jlzk7\" (UID: \"b3bd61c5-6cd4-434e-8b3d-487cbacf897f\") " pod="openshift-image-registry/node-ca-jlzk7"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303947    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303950    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-ovnkube-config\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.303993    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3bd61c5-6cd4-434e-8b3d-487cbacf897f-host\") pod \"node-ca-jlzk7\" (UID: \"b3bd61c5-6cd4-434e-8b3d-487cbacf897f\") " pod="openshift-image-registry/node-ca-jlzk7"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304035    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-systemd\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304062    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-run-ovn\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304086    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-kubernetes\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304087    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304111    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd84a8bd-6c02-477b-8e99-6b78466bea4a-kubelet-config\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304135    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-run-ovn\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304141    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlbzv\" (UniqueName: \"kubernetes.io/projected/b7a65ab8-59f7-4487-9329-7352324a3319-kube-api-access-qlbzv\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304147    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-systemd\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304168    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-run-openvswitch\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.304675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304187    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-kubernetes\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304206    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-run-openvswitch\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304205    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8b8w\" (UniqueName: \"kubernetes.io/projected/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-kube-api-access-t8b8w\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304233    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd84a8bd-6c02-477b-8e99-6b78466bea4a-kubelet-config\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304345    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-lib-modules\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304375    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-var-lib-kubelet\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304384    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3bd61c5-6cd4-434e-8b3d-487cbacf897f-serviceca\") pod \"node-ca-jlzk7\" (UID: \"b3bd61c5-6cd4-434e-8b3d-487cbacf897f\") " pod="openshift-image-registry/node-ca-jlzk7"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304402    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304411    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.304487    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304521    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-lib-modules\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304536    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-var-lib-kubelet\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.304550    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret podName:fd84a8bd-6c02-477b-8e99-6b78466bea4a nodeName:}" failed. No retries permitted until 2026-07-09 18:04:08.804531017 +0000 UTC m=+3.138395081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret") pod "global-pull-secret-syncer-pqd8h" (UID: "fd84a8bd-6c02-477b-8e99-6b78466bea4a") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304571    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-etc-selinux\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304599    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-node-log\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304642    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-node-log\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304672    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-log-socket\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.305531 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304701    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-ovnkube-script-lib\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304713    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-etc-selinux\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304721    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-log-socket\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304730    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304780    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-sysctl-conf\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304814    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304840    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-run-netns\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304868    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-run-ovn-kubernetes\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304914    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-systemd-units\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304941    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7rh\" (UniqueName: \"kubernetes.io/projected/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-kube-api-access-ln7rh\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304968    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-cni-binary-copy\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.304996    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-tuned\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305021    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-device-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305050    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-system-cni-dir\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305077    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-etc-openvswitch\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305104    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-var-lib-openvswitch\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305134    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-sysctl-d\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.306299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305160    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cda30b50-0c11-4e28-8ea9-a954ae1b8089-tmp\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305186    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10-hosts-file\") pod \"node-resolver-kg2kp\" (UID: \"24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10\") " pod="openshift-dns/node-resolver-kg2kp"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305203    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305211    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-sys-fs\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305238    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-os-release\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305273    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10-tmp-dir\") pod \"node-resolver-kg2kp\" (UID: \"24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10\") " pod="openshift-dns/node-resolver-kg2kp"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305275    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-run-netns\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305282    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-ovnkube-script-lib\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305344    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2wh\" (UniqueName: \"kubernetes.io/projected/24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10-kube-api-access-pk2wh\") pod \"node-resolver-kg2kp\" (UID: \"24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10\") " pod="openshift-dns/node-resolver-kg2kp"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305367    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-run-ovn-kubernetes\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305392    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-slash\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305422    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-sysctl-conf\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305425    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-systemd-units\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305465    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-var-lib-openvswitch\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305480    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-host-slash\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305492    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305547    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10-hosts-file\") pod \"node-resolver-kg2kp\" (UID: \"24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10\") " pod="openshift-dns/node-resolver-kg2kp"
Jul 09 18:04:08.307018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305659    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-sysctl-d\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.307651 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305799    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-cni-binary-copy\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.307651 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305806    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10-tmp-dir\") pod \"node-resolver-kg2kp\" (UID: \"24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10\") " pod="openshift-dns/node-resolver-kg2kp"
Jul 09 18:04:08.307651 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305902    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-sys-fs\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.307651 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305941    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-os-release\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.307651 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.305967    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-system-cni-dir\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.307651 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.306009    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b7a65ab8-59f7-4487-9329-7352324a3319-device-dir\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.307651 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.306015    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-etc-openvswitch\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.307651 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.307060    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-ovn-node-metrics-cert\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.308391 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.308308    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cda30b50-0c11-4e28-8ea9-a954ae1b8089-tmp\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.308391 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.308352    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cda30b50-0c11-4e28-8ea9-a954ae1b8089-etc-tuned\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.321309 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.321273    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvm6m\" (UniqueName: \"kubernetes.io/projected/b3bd61c5-6cd4-434e-8b3d-487cbacf897f-kube-api-access-bvm6m\") pod \"node-ca-jlzk7\" (UID: \"b3bd61c5-6cd4-434e-8b3d-487cbacf897f\") " pod="openshift-image-registry/node-ca-jlzk7"
Jul 09 18:04:08.323737 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.323678    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8b8w\" (UniqueName: \"kubernetes.io/projected/696c7d0f-9dad-4b5e-a4ab-081b032a3bf7-kube-api-access-t8b8w\") pod \"multus-additional-cni-plugins-npq7x\" (UID: \"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7\") " pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.323908 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.323886    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7rh\" (UniqueName: \"kubernetes.io/projected/73fdebde-0b8f-486b-aa5c-ef4b20aafffd-kube-api-access-ln7rh\") pod \"ovnkube-node-bstvh\" (UID: \"73fdebde-0b8f-486b-aa5c-ef4b20aafffd\") " pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.325000 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.324912    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlbzv\" (UniqueName: \"kubernetes.io/projected/b7a65ab8-59f7-4487-9329-7352324a3319-kube-api-access-qlbzv\") pod \"aws-ebs-csi-driver-node-hfkgb\" (UID: \"b7a65ab8-59f7-4487-9329-7352324a3319\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.326054 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.326030    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9lb4\" (UniqueName: \"kubernetes.io/projected/cda30b50-0c11-4e28-8ea9-a954ae1b8089-kube-api-access-t9lb4\") pod \"tuned-wvzlz\" (UID: \"cda30b50-0c11-4e28-8ea9-a954ae1b8089\") " pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.326425 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.326406    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2wh\" (UniqueName: \"kubernetes.io/projected/24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10-kube-api-access-pk2wh\") pod \"node-resolver-kg2kp\" (UID: \"24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10\") " pod="openshift-dns/node-resolver-kg2kp"
Jul 09 18:04:08.389488 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.389395    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bkn7f"
Jul 09 18:04:08.395866 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.395846    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:08.402887 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:08.402857    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4d5a34_f150_4a58_ad65_60b4c5f57295.slice/crio-09cef6d09ee321d1a1f4a3dd2057ec4b103c75f3f6f2b379e965b2d207723fa4 WatchSource:0}: Error finding container 09cef6d09ee321d1a1f4a3dd2057ec4b103c75f3f6f2b379e965b2d207723fa4: Status 404 returned error can't find the container with id 09cef6d09ee321d1a1f4a3dd2057ec4b103c75f3f6f2b379e965b2d207723fa4
Jul 09 18:04:08.404230 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.404206    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wvqdj"
Jul 09 18:04:08.411735 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:08.411708    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087b34f5_8d0f_4612_9f19_a87dd8fbf564.slice/crio-454e869e67e9d7bae6eb48336bd99c3655d3ad2cb310b82ea23a491da193fb33 WatchSource:0}: Error finding container 454e869e67e9d7bae6eb48336bd99c3655d3ad2cb310b82ea23a491da193fb33: Status 404 returned error can't find the container with id 454e869e67e9d7bae6eb48336bd99c3655d3ad2cb310b82ea23a491da193fb33
Jul 09 18:04:08.412030 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.412008    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb"
Jul 09 18:04:08.422665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.422637    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jlzk7"
Jul 09 18:04:08.427148 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:08.427111    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a65ab8_59f7_4487_9329_7352324a3319.slice/crio-2906feffa98d68dfa95b67781888587dbdaf8a2790a872a35ba16548a7548807 WatchSource:0}: Error finding container 2906feffa98d68dfa95b67781888587dbdaf8a2790a872a35ba16548a7548807: Status 404 returned error can't find the container with id 2906feffa98d68dfa95b67781888587dbdaf8a2790a872a35ba16548a7548807
Jul 09 18:04:08.430119 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.430097    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:08.435080 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:08.433585    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bd61c5_6cd4_434e_8b3d_487cbacf897f.slice/crio-bd5f37fca864d8263a52167f7fed49cccd3d291d1ba1a5d808cd69d93077c14b WatchSource:0}: Error finding container bd5f37fca864d8263a52167f7fed49cccd3d291d1ba1a5d808cd69d93077c14b: Status 404 returned error can't find the container with id bd5f37fca864d8263a52167f7fed49cccd3d291d1ba1a5d808cd69d93077c14b
Jul 09 18:04:08.435080 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.434862    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-npq7x"
Jul 09 18:04:08.442930 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.442752    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wvzlz"
Jul 09 18:04:08.445776 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:08.445669    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fdebde_0b8f_486b_aa5c_ef4b20aafffd.slice/crio-89b7b048bf97cd303446c82793c37f7edd470e099a05fd1b161f89f10078ea89 WatchSource:0}: Error finding container 89b7b048bf97cd303446c82793c37f7edd470e099a05fd1b161f89f10078ea89: Status 404 returned error can't find the container with id 89b7b048bf97cd303446c82793c37f7edd470e099a05fd1b161f89f10078ea89
Jul 09 18:04:08.449103 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:08.449063    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696c7d0f_9dad_4b5e_a4ab_081b032a3bf7.slice/crio-7f3b00849adb1c8734f8dab99586842227c513d0c5e9252c7da52343d54088b4 WatchSource:0}: Error finding container 7f3b00849adb1c8734f8dab99586842227c513d0c5e9252c7da52343d54088b4: Status 404 returned error can't find the container with id 7f3b00849adb1c8734f8dab99586842227c513d0c5e9252c7da52343d54088b4
Jul 09 18:04:08.455251 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.455098    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kg2kp"
Jul 09 18:04:08.457779 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:08.457748    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda30b50_0c11_4e28_8ea9_a954ae1b8089.slice/crio-26f861fd35bf032175e5f8efef4317ea52000d6e9ed4dc7222980fadf946363e WatchSource:0}: Error finding container 26f861fd35bf032175e5f8efef4317ea52000d6e9ed4dc7222980fadf946363e: Status 404 returned error can't find the container with id 26f861fd35bf032175e5f8efef4317ea52000d6e9ed4dc7222980fadf946363e
Jul 09 18:04:08.708661 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.708571    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:08.708862 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.708713    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:08.708862 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.708787    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs podName:c4b61885-5a4f-4355-ab6c-f95538b7992b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:09.708768165 +0000 UTC m=+4.042632230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs") pod "network-metrics-daemon-f84z2" (UID: "c4b61885-5a4f-4355-ab6c-f95538b7992b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:08.809548 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.809514    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:08.809732 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:08.809562    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc692\" (UniqueName: \"kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692\") pod \"network-check-target-bth88\" (UID: \"113c3ee7-a95e-42b6-87d8-68149f3382c8\") " pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:08.809732 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.809677    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:08.809732 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.809692    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:08.809732 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.809710    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:08.809732 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.809722    2568 projected.go:196] Error preparing data for projected volume kube-api-access-jc692 for pod openshift-network-diagnostics/network-check-target-bth88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:08.810036 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.809751    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret podName:fd84a8bd-6c02-477b-8e99-6b78466bea4a nodeName:}" failed. No retries permitted until 2026-07-09 18:04:09.809731718 +0000 UTC m=+4.143595798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret") pod "global-pull-secret-syncer-pqd8h" (UID: "fd84a8bd-6c02-477b-8e99-6b78466bea4a") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:08.810036 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:08.809769    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692 podName:113c3ee7-a95e-42b6-87d8-68149f3382c8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:09.809760368 +0000 UTC m=+4.143624433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jc692" (UniqueName: "kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692") pod "network-check-target-bth88" (UID: "113c3ee7-a95e-42b6-87d8-68149f3382c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:09.144491 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.144392    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:08 +0000 UTC" deadline="2027-12-17 22:42:50.245362155 +0000 UTC"
Jul 09 18:04:09.144491 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.144438    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12628h38m41.100926562s"
Jul 09 18:04:09.221463 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.221424    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:09.221651 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.221423    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:09.221651 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:09.221560    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:09.221651 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:09.221626    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:09.232381 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.232345    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-npq7x" event={"ID":"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7","Type":"ContainerStarted","Data":"7f3b00849adb1c8734f8dab99586842227c513d0c5e9252c7da52343d54088b4"}
Jul 09 18:04:09.233674 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.233641    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jlzk7" event={"ID":"b3bd61c5-6cd4-434e-8b3d-487cbacf897f","Type":"ContainerStarted","Data":"bd5f37fca864d8263a52167f7fed49cccd3d291d1ba1a5d808cd69d93077c14b"}
Jul 09 18:04:09.235032 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.235008    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb" event={"ID":"b7a65ab8-59f7-4487-9329-7352324a3319","Type":"ContainerStarted","Data":"2906feffa98d68dfa95b67781888587dbdaf8a2790a872a35ba16548a7548807"}
Jul 09 18:04:09.236305 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.236279    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wvqdj" event={"ID":"148bb254-e170-4306-9b3f-10811338d4b2","Type":"ContainerStarted","Data":"38703739a8b174612413de158643dd5aa6af0fb745052dc4fff44f1150f7792d"}
Jul 09 18:04:09.237454 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.237429    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xm9lj" event={"ID":"087b34f5-8d0f-4612-9f19-a87dd8fbf564","Type":"ContainerStarted","Data":"454e869e67e9d7bae6eb48336bd99c3655d3ad2cb310b82ea23a491da193fb33"}
Jul 09 18:04:09.238511 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.238482    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kg2kp" event={"ID":"24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10","Type":"ContainerStarted","Data":"818b54f610d86285a1993094cce1f575b617b020f3823c658a32c3764d39b935"}
Jul 09 18:04:09.239742 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.239718    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wvzlz" event={"ID":"cda30b50-0c11-4e28-8ea9-a954ae1b8089","Type":"ContainerStarted","Data":"26f861fd35bf032175e5f8efef4317ea52000d6e9ed4dc7222980fadf946363e"}
Jul 09 18:04:09.241176 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.241153    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh" event={"ID":"73fdebde-0b8f-486b-aa5c-ef4b20aafffd","Type":"ContainerStarted","Data":"89b7b048bf97cd303446c82793c37f7edd470e099a05fd1b161f89f10078ea89"}
Jul 09 18:04:09.242459 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.242429    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bkn7f" event={"ID":"ea4d5a34-f150-4a58-ad65-60b4c5f57295","Type":"ContainerStarted","Data":"09cef6d09ee321d1a1f4a3dd2057ec4b103c75f3f6f2b379e965b2d207723fa4"}
Jul 09 18:04:09.718834 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.718188    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:09.718834 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:09.718407    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:09.718834 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:09.718476    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs podName:c4b61885-5a4f-4355-ab6c-f95538b7992b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:11.718456832 +0000 UTC m=+6.052320908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs") pod "network-metrics-daemon-f84z2" (UID: "c4b61885-5a4f-4355-ab6c-f95538b7992b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:09.819585 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.819544    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:09.819780 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:09.819612    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc692\" (UniqueName: \"kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692\") pod \"network-check-target-bth88\" (UID: \"113c3ee7-a95e-42b6-87d8-68149f3382c8\") " pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:09.819780 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:09.819775    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:09.819904 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:09.819798    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:09.819904 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:09.819811    2568 projected.go:196] Error preparing data for projected volume kube-api-access-jc692 for pod openshift-network-diagnostics/network-check-target-bth88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:09.819904 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:09.819875    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692 podName:113c3ee7-a95e-42b6-87d8-68149f3382c8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:11.819852767 +0000 UTC m=+6.153716830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jc692" (UniqueName: "kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692") pod "network-check-target-bth88" (UID: "113c3ee7-a95e-42b6-87d8-68149f3382c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:09.820398 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:09.820370    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:09.820519 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:09.820438    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret podName:fd84a8bd-6c02-477b-8e99-6b78466bea4a nodeName:}" failed. No retries permitted until 2026-07-09 18:04:11.820421806 +0000 UTC m=+6.154285881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret") pod "global-pull-secret-syncer-pqd8h" (UID: "fd84a8bd-6c02-477b-8e99-6b78466bea4a") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:10.145035 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:10.144934    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:08 +0000 UTC" deadline="2028-01-20 00:40:19.127798505 +0000 UTC"
Jul 09 18:04:10.145035 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:10.144982    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13422h36m8.982822455s"
Jul 09 18:04:10.222090 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:10.222055    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:10.222310 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:10.222216    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:11.221498 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:11.221143    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:11.221498 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:11.221402    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:11.222022 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:11.221498    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:11.222022 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:11.221824    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:11.268160 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:11.268090    2568 generic.go:356] "Generic (PLEG): container finished" podID="46a34aa9fbfdc20a772f622b306ebd77" containerID="12845ebbfce493e3c4f7cc4693c128a7ca24d225850b1067fc105265e01c9f4c" exitCode=0
Jul 09 18:04:11.268331 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:11.268186    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal" event={"ID":"46a34aa9fbfdc20a772f622b306ebd77","Type":"ContainerDied","Data":"12845ebbfce493e3c4f7cc4693c128a7ca24d225850b1067fc105265e01c9f4c"}
Jul 09 18:04:11.277505 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:11.277475    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-234.ec2.internal" event={"ID":"27720bce3ffdbdb8ce1870f3c11220d9","Type":"ContainerStarted","Data":"fa38a859185273b852fbdcb635f76a0f64f9fe2e32ae3aae1ba06e337687f0ef"}
Jul 09 18:04:11.736063 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:11.736019    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:11.736264 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:11.736204    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:11.736343 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:11.736275    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs podName:c4b61885-5a4f-4355-ab6c-f95538b7992b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:15.736253364 +0000 UTC m=+10.070117446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs") pod "network-metrics-daemon-f84z2" (UID: "c4b61885-5a4f-4355-ab6c-f95538b7992b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:11.837860 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:11.837185    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc692\" (UniqueName: \"kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692\") pod \"network-check-target-bth88\" (UID: \"113c3ee7-a95e-42b6-87d8-68149f3382c8\") " pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:11.837860 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:11.837286    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:11.837860 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:11.837486    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:11.837860 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:11.837557    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret podName:fd84a8bd-6c02-477b-8e99-6b78466bea4a nodeName:}" failed. No retries permitted until 2026-07-09 18:04:15.837535643 +0000 UTC m=+10.171399708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret") pod "global-pull-secret-syncer-pqd8h" (UID: "fd84a8bd-6c02-477b-8e99-6b78466bea4a") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:11.837860 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:11.837683    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:11.837860 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:11.837705    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:11.837860 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:11.837718    2568 projected.go:196] Error preparing data for projected volume kube-api-access-jc692 for pod openshift-network-diagnostics/network-check-target-bth88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:11.837860 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:11.837767    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692 podName:113c3ee7-a95e-42b6-87d8-68149f3382c8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:15.837752799 +0000 UTC m=+10.171616875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jc692" (UniqueName: "kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692") pod "network-check-target-bth88" (UID: "113c3ee7-a95e-42b6-87d8-68149f3382c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:12.222178 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:12.222093    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:12.222702 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:12.222242    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:13.221666 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:13.221573    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:13.221666 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:13.221591    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:13.221969 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:13.221719    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:13.222035 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:13.222013    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:14.221910 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:14.221474    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:14.221910 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:14.221599    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:15.221297 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:15.221213    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:15.221297 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:15.221253    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:15.221516 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:15.221372    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:15.221835 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:15.221746    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:15.772691 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:15.772654    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:15.773089 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:15.772801    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:15.773089 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:15.772864    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs podName:c4b61885-5a4f-4355-ab6c-f95538b7992b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:23.772846319 +0000 UTC m=+18.106710384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs") pod "network-metrics-daemon-f84z2" (UID: "c4b61885-5a4f-4355-ab6c-f95538b7992b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:15.873753 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:15.873318    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:15.873753 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:15.873385    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc692\" (UniqueName: \"kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692\") pod \"network-check-target-bth88\" (UID: \"113c3ee7-a95e-42b6-87d8-68149f3382c8\") " pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:15.873753 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:15.873548    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:15.873753 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:15.873567    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:15.873753 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:15.873581    2568 projected.go:196] Error preparing data for projected volume kube-api-access-jc692 for pod openshift-network-diagnostics/network-check-target-bth88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:15.873753 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:15.873636    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692 podName:113c3ee7-a95e-42b6-87d8-68149f3382c8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:23.873617271 +0000 UTC m=+18.207481343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jc692" (UniqueName: "kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692") pod "network-check-target-bth88" (UID: "113c3ee7-a95e-42b6-87d8-68149f3382c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:15.873753 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:15.873700    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:15.873753 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:15.873732    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret podName:fd84a8bd-6c02-477b-8e99-6b78466bea4a nodeName:}" failed. No retries permitted until 2026-07-09 18:04:23.873721902 +0000 UTC m=+18.207585960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret") pod "global-pull-secret-syncer-pqd8h" (UID: "fd84a8bd-6c02-477b-8e99-6b78466bea4a") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:16.225574 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:16.225504    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:16.225726 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:16.225613    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:17.221742 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:17.221708    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:17.222215 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:17.221837    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:17.222299 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:17.222251    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:17.222393 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:17.222367    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:18.225244 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:18.225204    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:18.225748 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:18.225333    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:19.221045 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:19.221017    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:19.221196 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:19.221020    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:19.221196 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:19.221128    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:19.221267 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:19.221201    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:20.221733 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:20.221706    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:20.222096 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:20.221802    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:21.221822 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:21.221783    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:21.222294 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:21.221823    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:21.222294 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:21.221929    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:21.222294 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:21.222085    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:22.221632 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:22.221596    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:22.221820 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:22.221701    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:23.221703 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:23.221669    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:23.222184 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:23.221669    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:23.222184 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:23.221787    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:23.222184 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:23.221852    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:23.834596 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:23.834554    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:23.834903 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:23.834681    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:23.834903 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:23.834765    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs podName:c4b61885-5a4f-4355-ab6c-f95538b7992b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:39.834741943 +0000 UTC m=+34.168606003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs") pod "network-metrics-daemon-f84z2" (UID: "c4b61885-5a4f-4355-ab6c-f95538b7992b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:23.935459 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:23.935424    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc692\" (UniqueName: \"kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692\") pod \"network-check-target-bth88\" (UID: \"113c3ee7-a95e-42b6-87d8-68149f3382c8\") " pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:23.935625 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:23.935489    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:23.935625 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:23.935581    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:23.935625 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:23.935606    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:23.935719 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:23.935632    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:23.935719 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:23.935644    2568 projected.go:196] Error preparing data for projected volume kube-api-access-jc692 for pod openshift-network-diagnostics/network-check-target-bth88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:23.935719 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:23.935656    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret podName:fd84a8bd-6c02-477b-8e99-6b78466bea4a nodeName:}" failed. No retries permitted until 2026-07-09 18:04:39.935642466 +0000 UTC m=+34.269506521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret") pod "global-pull-secret-syncer-pqd8h" (UID: "fd84a8bd-6c02-477b-8e99-6b78466bea4a") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:23.935719 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:23.935687    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692 podName:113c3ee7-a95e-42b6-87d8-68149f3382c8 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:39.935675916 +0000 UTC m=+34.269539972 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jc692" (UniqueName: "kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692") pod "network-check-target-bth88" (UID: "113c3ee7-a95e-42b6-87d8-68149f3382c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:24.221468 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:24.221390    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:24.221624 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:24.221516    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:25.221686 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:25.221649    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:25.221686 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:25.221674    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:25.222150 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:25.221775    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:25.222150 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:25.221912    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:26.221960 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:26.221918    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:26.222385 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:26.222035    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:27.222021 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:27.221982    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:27.222437 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:27.221989    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:27.222437 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:27.222128    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:27.222437 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:27.222189    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:28.221762 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:28.221732    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:28.221929 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:28.221841    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:29.221526 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.221302    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:29.222115 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.221339    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:29.222115 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:29.221638    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:29.222115 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:29.221695    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:29.317452 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.317262    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb" event={"ID":"b7a65ab8-59f7-4487-9329-7352324a3319","Type":"ContainerStarted","Data":"240bbcc3c5050d003404fd07833b0401a42f20235a38ad3791c95a906179399d"}
Jul 09 18:04:29.318746 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.318722    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wvqdj" event={"ID":"148bb254-e170-4306-9b3f-10811338d4b2","Type":"ContainerStarted","Data":"27a6462b30c4b614e4dcf73f7ed7dca3c8dd141d356405c0c3d4c3169e2afe08"}
Jul 09 18:04:29.320156 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.320136    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xm9lj" event={"ID":"087b34f5-8d0f-4612-9f19-a87dd8fbf564","Type":"ContainerStarted","Data":"f2468946f1f5e31955bb1cbc262aa316f2bda48ca872e2e6113a060ce2348d81"}
Jul 09 18:04:29.321578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.321545    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kg2kp" event={"ID":"24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10","Type":"ContainerStarted","Data":"a8373ac1c5c1bf19a700b02f1601ac9b2b9b012f88c141f76090a4447aceec88"}
Jul 09 18:04:29.323149 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.323128    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wvzlz" event={"ID":"cda30b50-0c11-4e28-8ea9-a954ae1b8089","Type":"ContainerStarted","Data":"b940e661d06cfd739a591d5512ed56785d6f75c26161485585548c73764ab74a"}
Jul 09 18:04:29.325866 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.325847    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:04:29.326246 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.326218    2568 generic.go:356] "Generic (PLEG): container finished" podID="73fdebde-0b8f-486b-aa5c-ef4b20aafffd" containerID="c06b3e3141b78d56399a5fcbc21b6da0e2e1c6865b83ad2df79d5d5ebb3dd672" exitCode=1
Jul 09 18:04:29.326348 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.326287    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh" event={"ID":"73fdebde-0b8f-486b-aa5c-ef4b20aafffd","Type":"ContainerStarted","Data":"181169008f8e6618137b6aa33b19ca85c95e7c2982dbf3eb43061f41457f4d16"}
Jul 09 18:04:29.326405 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.326350    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh" event={"ID":"73fdebde-0b8f-486b-aa5c-ef4b20aafffd","Type":"ContainerStarted","Data":"40e44a020ce2b1082a33e98c7e81a47e4eb3875a17d8183544156ec1479d9294"}
Jul 09 18:04:29.326405 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.326399    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh" event={"ID":"73fdebde-0b8f-486b-aa5c-ef4b20aafffd","Type":"ContainerStarted","Data":"527744e93749a6cf2b5191b6058483aeb032d453d1db23a0b789f3fba476721f"}
Jul 09 18:04:29.326495 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.326445    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh" event={"ID":"73fdebde-0b8f-486b-aa5c-ef4b20aafffd","Type":"ContainerStarted","Data":"fac2478f560712f09c43d2ac75b99c23f1ea03274bda5da5dbc56bc1e438a8f5"}
Jul 09 18:04:29.326544 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.326494    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh" event={"ID":"73fdebde-0b8f-486b-aa5c-ef4b20aafffd","Type":"ContainerDied","Data":"c06b3e3141b78d56399a5fcbc21b6da0e2e1c6865b83ad2df79d5d5ebb3dd672"}
Jul 09 18:04:29.326590 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.326546    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh" event={"ID":"73fdebde-0b8f-486b-aa5c-ef4b20aafffd","Type":"ContainerStarted","Data":"02e7ebd09683859074ed4875d496437dd608a7a4ca19e2723d4fbe3fcad62716"}
Jul 09 18:04:29.328517 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.328469    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal" event={"ID":"46a34aa9fbfdc20a772f622b306ebd77","Type":"ContainerStarted","Data":"14cac6761e1ba8050def9a8e63f85172772f1099151f5581df3f15af60c85ed8"}
Jul 09 18:04:29.330115 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.330095    2568 generic.go:356] "Generic (PLEG): container finished" podID="696c7d0f-9dad-4b5e-a4ab-081b032a3bf7" containerID="5485dad6e3249acf7c2c894c1267ac1bf27ef7320f85923b047b33e078029a40" exitCode=0
Jul 09 18:04:29.330187 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.330162    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-npq7x" event={"ID":"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7","Type":"ContainerDied","Data":"5485dad6e3249acf7c2c894c1267ac1bf27ef7320f85923b047b33e078029a40"}
Jul 09 18:04:29.331632 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.331556    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jlzk7" event={"ID":"b3bd61c5-6cd4-434e-8b3d-487cbacf897f","Type":"ContainerStarted","Data":"c4f77dcc94ae5d11df74c7c4954841341801a5605d4a6d9106c6307b457328af"}
Jul 09 18:04:29.348459 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.348360    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-234.ec2.internal" podStartSLOduration=22.348346926 podStartE2EDuration="22.348346926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:11.313604068 +0000 UTC m=+5.647468148" watchObservedRunningTime="2026-07-09 18:04:29.348346926 +0000 UTC m=+23.682211004"
Jul 09 18:04:29.348632 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.348604    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wvqdj" podStartSLOduration=3.103353396 podStartE2EDuration="23.348596257s" totalImagesPullingTime="20.245242861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:29.347258725 +0000 UTC m=+23.681122804" watchObservedRunningTime="2026-07-09 18:04:29.348596257 +0000 UTC m=+23.682460335"
Jul 09 18:04:29.397763 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.397720    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jlzk7" podStartSLOduration=3.198127634 podStartE2EDuration="23.397705714s" totalImagesPullingTime="20.19957808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:29.366429763 +0000 UTC m=+23.700293842" watchObservedRunningTime="2026-07-09 18:04:29.397705714 +0000 UTC m=+23.731569792"
Jul 09 18:04:29.421620 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.421577    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kg2kp" podStartSLOduration=2.248146233 podStartE2EDuration="22.421562638s" totalImagesPullingTime="20.173416405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:29.421260111 +0000 UTC m=+23.755124192" watchObservedRunningTime="2026-07-09 18:04:29.421562638 +0000 UTC m=+23.755426715"
Jul 09 18:04:29.475198 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.475155    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-234.ec2.internal" podStartSLOduration=22.475139468 podStartE2EDuration="22.475139468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:29.474712707 +0000 UTC m=+23.808576784" watchObservedRunningTime="2026-07-09 18:04:29.475139468 +0000 UTC m=+23.809003547"
Jul 09 18:04:29.475572 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.475547    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wvzlz" podStartSLOduration=3.291900586 podStartE2EDuration="23.475539933s" totalImagesPullingTime="20.183639347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:29.449801151 +0000 UTC m=+23.783665242" watchObservedRunningTime="2026-07-09 18:04:29.475539933 +0000 UTC m=+23.809404011"
Jul 09 18:04:29.517732 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:29.517679    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xm9lj" podStartSLOduration=3.311933146 podStartE2EDuration="23.517662382s" totalImagesPullingTime="20.205729236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:29.516937705 +0000 UTC m=+23.850801784" watchObservedRunningTime="2026-07-09 18:04:29.517662382 +0000 UTC m=+23.851526462"
Jul 09 18:04:30.221691 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:30.221660    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:30.222158 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:30.221796    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:30.342546 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:30.341479    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bkn7f" event={"ID":"ea4d5a34-f150-4a58-ad65-60b4c5f57295","Type":"ContainerStarted","Data":"7a31d5f38573221554b6e3e9547ca13d6f33012a1890a5639594e000ecf4040c"}
Jul 09 18:04:30.342858 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:30.342836    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:04:30.361635 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:30.361585    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bkn7f" podStartSLOduration=4.145761767 podStartE2EDuration="24.361564453s" totalImagesPullingTime="20.215802686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:30.36102175 +0000 UTC m=+24.694885828" watchObservedRunningTime="2026-07-09 18:04:30.361564453 +0000 UTC m=+24.695428531"
Jul 09 18:04:31.180058 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:31.179937    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:04:30.342852127Z","UUID":"6d64d351-d519-4d55-b663-68bd74f1f4c9","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:04:31.181557 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:31.181532    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:04:31.181694 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:31.181573    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:04:31.221703 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:31.221653    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:31.221703 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:31.221670    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:31.222222 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:31.221767    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:31.222222 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:31.221875    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:31.345161 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:31.345123    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb" event={"ID":"b7a65ab8-59f7-4487-9329-7352324a3319","Type":"ContainerStarted","Data":"5c64ff46966e62f42e13b5ad6c0fd01b55256c83cbdef15346dd572a9878a7ad"}
Jul 09 18:04:32.221425 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:32.221206    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:32.221611 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:32.221532    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:32.349723 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:32.349687    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb" event={"ID":"b7a65ab8-59f7-4487-9329-7352324a3319","Type":"ContainerStarted","Data":"17fad67650d47c1c2ed4cdd8ea0254fc3a93bbacdb167eb6f7e9ceb9cb6f7d51"}
Jul 09 18:04:32.353102 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:32.353078    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:04:32.353524 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:32.353499    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh" event={"ID":"73fdebde-0b8f-486b-aa5c-ef4b20aafffd","Type":"ContainerStarted","Data":"5e0db6417a18807937b3dd9f0e4639347379efa7269fa81f3420108e1ac6d77e"}
Jul 09 18:04:32.369851 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:32.369811    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfkgb" podStartSLOduration=3.431569797 podStartE2EDuration="26.369795852s" totalImagesPullingTime="22.938226055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:32.369598403 +0000 UTC m=+26.703462483" watchObservedRunningTime="2026-07-09 18:04:32.369795852 +0000 UTC m=+26.703659930"
Jul 09 18:04:33.221801 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:33.221771    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:33.221989 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:33.221898    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:33.221989 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:33.221931    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:33.222097 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:33.222060    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:33.394185 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:33.394156    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:33.394756 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:33.394741    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:34.221689 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:34.221651    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:34.221840 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:34.221783    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:34.359495 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:34.359460    2568 generic.go:356] "Generic (PLEG): container finished" podID="696c7d0f-9dad-4b5e-a4ab-081b032a3bf7" containerID="9b3033b4daac40e34a40626e12513e40b4963e9a13c5d2536d1f052d67ea6d6e" exitCode=0
Jul 09 18:04:34.359667 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:34.359553    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-npq7x" event={"ID":"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7","Type":"ContainerDied","Data":"9b3033b4daac40e34a40626e12513e40b4963e9a13c5d2536d1f052d67ea6d6e"}
Jul 09 18:04:34.363064 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:34.363046    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:04:34.363441 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:34.363417    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh" event={"ID":"73fdebde-0b8f-486b-aa5c-ef4b20aafffd","Type":"ContainerStarted","Data":"7d89fdf4a1cac004862b65fce040fbe4d8b9099ab734303a84d68a9a5b0dcdd7"}
Jul 09 18:04:34.363615 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:34.363599    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:34.363977 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:34.363953    2568 scope.go:117] "RemoveContainer" containerID="c06b3e3141b78d56399a5fcbc21b6da0e2e1c6865b83ad2df79d5d5ebb3dd672"
Jul 09 18:04:34.364138 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:34.364125    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xm9lj"
Jul 09 18:04:35.221590 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.221560    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:35.222283 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.221596    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:35.222283 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:35.221698    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:35.222283 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:35.221812    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:35.369753 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.369731    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:04:35.370192 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.370164    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh" event={"ID":"73fdebde-0b8f-486b-aa5c-ef4b20aafffd","Type":"ContainerStarted","Data":"f2393efbbc0c2260b8b7abb0164fe7979c3c65f944fb9190e16982956ef2b269"}
Jul 09 18:04:35.370742 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.370371    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:35.370742 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.370442    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:35.370742 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.370492    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:35.386413 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.386387    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:35.386561 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.386492    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:04:35.404842 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.404800    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh" podStartSLOduration=9.14545706 podStartE2EDuration="29.404787602s" totalImagesPullingTime="20.259330542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:35.404639784 +0000 UTC m=+29.738503881" watchObservedRunningTime="2026-07-09 18:04:35.404787602 +0000 UTC m=+29.738651680"
Jul 09 18:04:35.700942 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.700752    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bth88"]
Jul 09 18:04:35.701144 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.701021    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:35.701144 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:35.701097    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:35.702856 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.702830    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pqd8h"]
Jul 09 18:04:35.702980 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.702938    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:35.703065 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:35.703049    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:35.708478 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.708453    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f84z2"]
Jul 09 18:04:35.708569 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:35.708557    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:35.708684 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:35.708662    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:36.373693 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:36.373658    2568 generic.go:356] "Generic (PLEG): container finished" podID="696c7d0f-9dad-4b5e-a4ab-081b032a3bf7" containerID="ceede163c874cb8d37969fa9fb184472effb3f3321a1b6acff3129831507c97f" exitCode=0
Jul 09 18:04:36.374154 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:36.373741    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-npq7x" event={"ID":"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7","Type":"ContainerDied","Data":"ceede163c874cb8d37969fa9fb184472effb3f3321a1b6acff3129831507c97f"}
Jul 09 18:04:37.221692 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:37.221658    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:37.221877 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:37.221703    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:37.221877 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:37.221761    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:37.221877 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:37.221781    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:37.221877 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:37.221836    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:37.222131 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:37.221878    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:38.380772 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:38.380745    2568 generic.go:356] "Generic (PLEG): container finished" podID="696c7d0f-9dad-4b5e-a4ab-081b032a3bf7" containerID="c3288ebf74cb6f70a53129be1126b8cc6ac26d6fc313311321bb7412fcff21cf" exitCode=0
Jul 09 18:04:38.381501 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:38.380785    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-npq7x" event={"ID":"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7","Type":"ContainerDied","Data":"c3288ebf74cb6f70a53129be1126b8cc6ac26d6fc313311321bb7412fcff21cf"}
Jul 09 18:04:39.221847 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:39.221808    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:39.221847 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:39.221840    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:39.222155 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:39.221869    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:39.222155 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:39.221990    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:39.222155 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:39.222122    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:39.222283 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:39.222241    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:39.855083 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:39.855045    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:39.855473 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:39.855185    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:39.855473 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:39.855250    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs podName:c4b61885-5a4f-4355-ab6c-f95538b7992b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:11.855231922 +0000 UTC m=+66.189095998 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs") pod "network-metrics-daemon-f84z2" (UID: "c4b61885-5a4f-4355-ab6c-f95538b7992b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:39.956307 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:39.956238    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc692\" (UniqueName: \"kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692\") pod \"network-check-target-bth88\" (UID: \"113c3ee7-a95e-42b6-87d8-68149f3382c8\") " pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:39.956307 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:39.956293    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:39.956492 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:39.956375    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:39.956492 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:39.956389    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:39.956492 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:39.956409    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:39.956492 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:39.956418    2568 projected.go:196] Error preparing data for projected volume kube-api-access-jc692 for pod openshift-network-diagnostics/network-check-target-bth88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:39.956492 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:39.956421    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret podName:fd84a8bd-6c02-477b-8e99-6b78466bea4a nodeName:}" failed. No retries permitted until 2026-07-09 18:05:11.956407002 +0000 UTC m=+66.290271059 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret") pod "global-pull-secret-syncer-pqd8h" (UID: "fd84a8bd-6c02-477b-8e99-6b78466bea4a") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:39.956492 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:39.956455    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692 podName:113c3ee7-a95e-42b6-87d8-68149f3382c8 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:11.956443751 +0000 UTC m=+66.290307807 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jc692" (UniqueName: "kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692") pod "network-check-target-bth88" (UID: "113c3ee7-a95e-42b6-87d8-68149f3382c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:41.221705 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.221675    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:41.221705 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.221705    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:41.222205 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.221705    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:41.222205 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:41.221820    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:04:41.222205 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:41.221919    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqd8h" podUID="fd84a8bd-6c02-477b-8e99-6b78466bea4a"
Jul 09 18:04:41.222205 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:41.221990    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bth88" podUID="113c3ee7-a95e-42b6-87d8-68149f3382c8"
Jul 09 18:04:41.361718 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.361686    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-234.ec2.internal" event="NodeReady"
Jul 09 18:04:41.361897 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.361838    2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:04:41.441294 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.441190    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-46xvq"]
Jul 09 18:04:41.443463 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.443431    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.446999 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.446973    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:04:41.447691 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.447551    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:04:41.447691 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.447669    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jd75b\""
Jul 09 18:04:41.448718 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.448698    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gjkgm"]
Jul 09 18:04:41.450564 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.450544    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gjkgm"
Jul 09 18:04:41.456938 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.456918    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:04:41.457094 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.457057    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.457502 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.457162    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:04:41.457502 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.457330    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lkb9p\""
Jul 09 18:04:41.468111 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.468087    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-46xvq"]
Jul 09 18:04:41.468814 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.468791    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gjkgm"]
Jul 09 18:04:41.568506 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.568465    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dswf\" (UniqueName: \"kubernetes.io/projected/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-kube-api-access-4dswf\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.568506 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.568522    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgq9s\" (UniqueName: \"kubernetes.io/projected/1cae8e1a-1640-4639-8783-74995450b5a3-kube-api-access-rgq9s\") pod \"ingress-canary-gjkgm\" (UID: \"1cae8e1a-1640-4639-8783-74995450b5a3\") " pod="openshift-ingress-canary/ingress-canary-gjkgm"
Jul 09 18:04:41.568740 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.568559    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.568740 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.568617    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-config-volume\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.568740 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.568645    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-tmp-dir\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.568740 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.568693    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cae8e1a-1640-4639-8783-74995450b5a3-cert\") pod \"ingress-canary-gjkgm\" (UID: \"1cae8e1a-1640-4639-8783-74995450b5a3\") " pod="openshift-ingress-canary/ingress-canary-gjkgm"
Jul 09 18:04:41.669660 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.669620    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cae8e1a-1640-4639-8783-74995450b5a3-cert\") pod \"ingress-canary-gjkgm\" (UID: \"1cae8e1a-1640-4639-8783-74995450b5a3\") " pod="openshift-ingress-canary/ingress-canary-gjkgm"
Jul 09 18:04:41.669842 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.669698    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dswf\" (UniqueName: \"kubernetes.io/projected/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-kube-api-access-4dswf\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.669842 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.669731    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgq9s\" (UniqueName: \"kubernetes.io/projected/1cae8e1a-1640-4639-8783-74995450b5a3-kube-api-access-rgq9s\") pod \"ingress-canary-gjkgm\" (UID: \"1cae8e1a-1640-4639-8783-74995450b5a3\") " pod="openshift-ingress-canary/ingress-canary-gjkgm"
Jul 09 18:04:41.669842 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.669759    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.669842 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.669785    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-config-volume\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.669842 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.669806    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-tmp-dir\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.670149 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.670124    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-tmp-dir\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.670230 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:41.670204    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:41.670285 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:41.670254    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls podName:2b91a8a8-43dc-4f19-b55c-62e2b06801f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.170238958 +0000 UTC m=+36.504103016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls") pod "dns-default-46xvq" (UID: "2b91a8a8-43dc-4f19-b55c-62e2b06801f2") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:41.670675 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.670647    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-config-volume\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.674500 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.674480    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cae8e1a-1640-4639-8783-74995450b5a3-cert\") pod \"ingress-canary-gjkgm\" (UID: \"1cae8e1a-1640-4639-8783-74995450b5a3\") " pod="openshift-ingress-canary/ingress-canary-gjkgm"
Jul 09 18:04:41.682427 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.682394    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dswf\" (UniqueName: \"kubernetes.io/projected/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-kube-api-access-4dswf\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:41.683269 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.683245    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgq9s\" (UniqueName: \"kubernetes.io/projected/1cae8e1a-1640-4639-8783-74995450b5a3-kube-api-access-rgq9s\") pod \"ingress-canary-gjkgm\" (UID: \"1cae8e1a-1640-4639-8783-74995450b5a3\") " pod="openshift-ingress-canary/ingress-canary-gjkgm"
Jul 09 18:04:41.762424 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:41.762394    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gjkgm"
Jul 09 18:04:42.172744 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:42.172659    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:42.172908 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:42.172806    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:42.172908 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:42.172864    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls podName:2b91a8a8-43dc-4f19-b55c-62e2b06801f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:43.172846475 +0000 UTC m=+37.506710537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls") pod "dns-default-46xvq" (UID: "2b91a8a8-43dc-4f19-b55c-62e2b06801f2") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:42.680534 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:42.680494    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gjkgm"]
Jul 09 18:04:42.684941 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:04:42.684919    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cae8e1a_1640_4639_8783_74995450b5a3.slice/crio-8e000421ee952f246bb032a4068b9aa993a09c28c32189fc67bc5fd040cc8bcc WatchSource:0}: Error finding container 8e000421ee952f246bb032a4068b9aa993a09c28c32189fc67bc5fd040cc8bcc: Status 404 returned error can't find the container with id 8e000421ee952f246bb032a4068b9aa993a09c28c32189fc67bc5fd040cc8bcc
Jul 09 18:04:43.180654 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.180623    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:43.180824 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:43.180745    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:43.180824 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:43.180796    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls podName:2b91a8a8-43dc-4f19-b55c-62e2b06801f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:45.180784213 +0000 UTC m=+39.514648269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls") pod "dns-default-46xvq" (UID: "2b91a8a8-43dc-4f19-b55c-62e2b06801f2") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:43.221455 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.221418    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:04:43.221580 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.221476    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:04:43.221580 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.221435    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:04:43.225093 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.225074    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:04:43.225461 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.225447    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:04:43.226346 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.226329    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:04:43.226452 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.226369    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7k8ns\""
Jul 09 18:04:43.226674 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.226651    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:04:43.226674 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.226668    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w9bx9\""
Jul 09 18:04:43.396319 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.396109    2568 generic.go:356] "Generic (PLEG): container finished" podID="696c7d0f-9dad-4b5e-a4ab-081b032a3bf7" containerID="7e5741ac9abcb3055ad6ac192339cfacf8f0c1d08ad0708186321f7060625196" exitCode=0
Jul 09 18:04:43.396499 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.396197    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-npq7x" event={"ID":"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7","Type":"ContainerDied","Data":"7e5741ac9abcb3055ad6ac192339cfacf8f0c1d08ad0708186321f7060625196"}
Jul 09 18:04:43.397488 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:43.397464    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gjkgm" event={"ID":"1cae8e1a-1640-4639-8783-74995450b5a3","Type":"ContainerStarted","Data":"8e000421ee952f246bb032a4068b9aa993a09c28c32189fc67bc5fd040cc8bcc"}
Jul 09 18:04:44.403692 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:44.403655    2568 generic.go:356] "Generic (PLEG): container finished" podID="696c7d0f-9dad-4b5e-a4ab-081b032a3bf7" containerID="d9240d90113cd37c6b90d6e51871d431656ce1796f8021b17226efc846a57488" exitCode=0
Jul 09 18:04:44.404172 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:44.403714    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-npq7x" event={"ID":"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7","Type":"ContainerDied","Data":"d9240d90113cd37c6b90d6e51871d431656ce1796f8021b17226efc846a57488"}
Jul 09 18:04:45.195714 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:45.195634    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:45.195871 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:45.195801    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:45.195871 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:45.195865    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls podName:2b91a8a8-43dc-4f19-b55c-62e2b06801f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:49.195849587 +0000 UTC m=+43.529713644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls") pod "dns-default-46xvq" (UID: "2b91a8a8-43dc-4f19-b55c-62e2b06801f2") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:45.409338 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:45.409306    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-npq7x" event={"ID":"696c7d0f-9dad-4b5e-a4ab-081b032a3bf7","Type":"ContainerStarted","Data":"c8470bfe8d186c1217a3452e43bae58755d153075a216e7201c1797eb88d9f49"}
Jul 09 18:04:45.410760 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:45.410737    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gjkgm" event={"ID":"1cae8e1a-1640-4639-8783-74995450b5a3","Type":"ContainerStarted","Data":"c7fc9670d4cdfcf4b424b99b78e2d832bf3bfa98b893d4aa78f9cc27c0a5bfb3"}
Jul 09 18:04:45.435043 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:45.434996    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-npq7x" podStartSLOduration=8.658805481 podStartE2EDuration="39.434978877s" totalImagesPullingTime="30.776173396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:45.433005869 +0000 UTC m=+39.766869948" watchObservedRunningTime="2026-07-09 18:04:45.434978877 +0000 UTC m=+39.768842955"
Jul 09 18:04:45.449401 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:45.449323    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gjkgm" podStartSLOduration=2.121081291 podStartE2EDuration="4.449310623s" totalImagesPullingTime="2.328229332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:45.448847096 +0000 UTC m=+39.782711200" watchObservedRunningTime="2026-07-09 18:04:45.449310623 +0000 UTC m=+39.783174727"
Jul 09 18:04:49.222742 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:49.222692    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:49.223230 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:49.222848    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:49.223230 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:49.222931    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls podName:2b91a8a8-43dc-4f19-b55c-62e2b06801f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:57.222914313 +0000 UTC m=+51.556778368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls") pod "dns-default-46xvq" (UID: "2b91a8a8-43dc-4f19-b55c-62e2b06801f2") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:57.274235 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:04:57.274197    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:04:57.274712 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:57.274341    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:57.274712 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:04:57.274406    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls podName:2b91a8a8-43dc-4f19-b55c-62e2b06801f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:13.274391098 +0000 UTC m=+67.608255154 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls") pod "dns-default-46xvq" (UID: "2b91a8a8-43dc-4f19-b55c-62e2b06801f2") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:07.392384 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:07.392357    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bstvh"
Jul 09 18:05:11.872030 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:11.871979    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:05:11.874897 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:11.874867    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:11.882378 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:05:11.882356    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 18:05:11.882458 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:05:11.882424    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs podName:c4b61885-5a4f-4355-ab6c-f95538b7992b nodeName:}" failed. No retries permitted until 2026-07-09 18:06:15.882408048 +0000 UTC m=+130.216272103 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs") pod "network-metrics-daemon-f84z2" (UID: "c4b61885-5a4f-4355-ab6c-f95538b7992b") : secret "metrics-daemon-secret" not found
Jul 09 18:05:11.972360 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:11.972332    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:05:11.972448 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:11.972382    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc692\" (UniqueName: \"kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692\") pod \"network-check-target-bth88\" (UID: \"113c3ee7-a95e-42b6-87d8-68149f3382c8\") " pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:05:11.975151 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:11.975129    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:11.975246 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:11.975200    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:11.985640 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:11.985618    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:11.986538 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:11.986521    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd84a8bd-6c02-477b-8e99-6b78466bea4a-original-pull-secret\") pod \"global-pull-secret-syncer-pqd8h\" (UID: \"fd84a8bd-6c02-477b-8e99-6b78466bea4a\") " pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:05:11.995665 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:11.995646    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc692\" (UniqueName: \"kubernetes.io/projected/113c3ee7-a95e-42b6-87d8-68149f3382c8-kube-api-access-jc692\") pod \"network-check-target-bth88\" (UID: \"113c3ee7-a95e-42b6-87d8-68149f3382c8\") " pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:05:12.030720 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:12.030689    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqd8h"
Jul 09 18:05:12.040120 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:12.040099    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w9bx9\""
Jul 09 18:05:12.047829 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:12.047808    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:05:12.157684 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:12.157652    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pqd8h"]
Jul 09 18:05:12.160816 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:05:12.160787    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd84a8bd_6c02_477b_8e99_6b78466bea4a.slice/crio-c5b7c83ecb337881ae08d35b81b36624827e52da74bf255028173acf9fb8b8fa WatchSource:0}: Error finding container c5b7c83ecb337881ae08d35b81b36624827e52da74bf255028173acf9fb8b8fa: Status 404 returned error can't find the container with id c5b7c83ecb337881ae08d35b81b36624827e52da74bf255028173acf9fb8b8fa
Jul 09 18:05:12.380153 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:12.380067    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bth88"]
Jul 09 18:05:12.384238 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:05:12.384212    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113c3ee7_a95e_42b6_87d8_68149f3382c8.slice/crio-c1b900f938a8d1eba7ebda5f2e3375aba72c88482ff48d6d7ba541d4c99abe59 WatchSource:0}: Error finding container c1b900f938a8d1eba7ebda5f2e3375aba72c88482ff48d6d7ba541d4c99abe59: Status 404 returned error can't find the container with id c1b900f938a8d1eba7ebda5f2e3375aba72c88482ff48d6d7ba541d4c99abe59
Jul 09 18:05:12.470583 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:12.470549    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bth88" event={"ID":"113c3ee7-a95e-42b6-87d8-68149f3382c8","Type":"ContainerStarted","Data":"c1b900f938a8d1eba7ebda5f2e3375aba72c88482ff48d6d7ba541d4c99abe59"}
Jul 09 18:05:12.471700 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:12.471678    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pqd8h" event={"ID":"fd84a8bd-6c02-477b-8e99-6b78466bea4a","Type":"ContainerStarted","Data":"c5b7c83ecb337881ae08d35b81b36624827e52da74bf255028173acf9fb8b8fa"}
Jul 09 18:05:13.290779 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:13.290738    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:05:13.291208 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:05:13.290901    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:13.291208 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:05:13.290969    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls podName:2b91a8a8-43dc-4f19-b55c-62e2b06801f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:45.290948933 +0000 UTC m=+99.624813004 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls") pod "dns-default-46xvq" (UID: "2b91a8a8-43dc-4f19-b55c-62e2b06801f2") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:17.485869 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.485776    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bth88" event={"ID":"113c3ee7-a95e-42b6-87d8-68149f3382c8","Type":"ContainerStarted","Data":"c9a9d01c6e83fe1742d555a0e0598869ee6868f967a23bd099491b28bc677158"}
Jul 09 18:05:17.486321 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.485938    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:05:17.487159 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.487134    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pqd8h" event={"ID":"fd84a8bd-6c02-477b-8e99-6b78466bea4a","Type":"ContainerStarted","Data":"8ffe89ec8f70c74bf86fadeeb0ff231b178931297267176ec9ab423057a79773"}
Jul 09 18:05:17.501189 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.501147    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bth88" podStartSLOduration=66.86297121 podStartE2EDuration="1m11.501136505s" totalImagesPullingTime="4.638165295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:17.501013227 +0000 UTC m=+71.834877304" watchObservedRunningTime="2026-07-09 18:05:17.501136505 +0000 UTC m=+71.835000603"
Jul 09 18:05:17.601225 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.601180    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pqd8h" podStartSLOduration=65.738067756 podStartE2EDuration="1m10.601164961s" totalImagesPullingTime="4.863097205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:17.515967127 +0000 UTC m=+71.849831228" watchObservedRunningTime="2026-07-09 18:05:17.601164961 +0000 UTC m=+71.935029041"
Jul 09 18:05:17.601517 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.601495    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"]
Jul 09 18:05:17.603165 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.603151    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:17.606080 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.606055    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 18:05:17.606080 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.606074    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 18:05:17.606262 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.606109    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 18:05:17.606262 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.606140    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 18:05:17.625424 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.625400    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"]
Jul 09 18:05:17.627332 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.627307    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"]
Jul 09 18:05:17.627431 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.627410    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.630720 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.630701    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 18:05:17.630939 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.630921    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 18:05:17.631034 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.630958    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 18:05:17.631095 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.631083    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 18:05:17.646428 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.646396    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"]
Jul 09 18:05:17.726367 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.726342    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2e5749b0-4658-447f-9b33-26385f31990a-klusterlet-config\") pod \"klusterlet-addon-workmgr-69478447dc-2jscf\" (UID: \"2e5749b0-4658-447f-9b33-26385f31990a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:17.726498 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.726382    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.726498 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.726428    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6xf\" (UniqueName: \"kubernetes.io/projected/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-kube-api-access-qm6xf\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.726498 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.726449    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-hub\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.726607 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.726497    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e5749b0-4658-447f-9b33-26385f31990a-tmp\") pod \"klusterlet-addon-workmgr-69478447dc-2jscf\" (UID: \"2e5749b0-4658-447f-9b33-26385f31990a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:17.726607 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.726517    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzcwv\" (UniqueName: \"kubernetes.io/projected/2e5749b0-4658-447f-9b33-26385f31990a-kube-api-access-bzcwv\") pod \"klusterlet-addon-workmgr-69478447dc-2jscf\" (UID: \"2e5749b0-4658-447f-9b33-26385f31990a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:17.726607 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.726538    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-ca\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.726607 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.726554    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.726607 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.726581    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.827660 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.827631    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.827832 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.827675    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.827832 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.827708    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2e5749b0-4658-447f-9b33-26385f31990a-klusterlet-config\") pod \"klusterlet-addon-workmgr-69478447dc-2jscf\" (UID: \"2e5749b0-4658-447f-9b33-26385f31990a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:17.827832 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.827746    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.827832 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.827777    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6xf\" (UniqueName: \"kubernetes.io/projected/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-kube-api-access-qm6xf\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.827832 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.827800    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-hub\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.828114 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.827852    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e5749b0-4658-447f-9b33-26385f31990a-tmp\") pod \"klusterlet-addon-workmgr-69478447dc-2jscf\" (UID: \"2e5749b0-4658-447f-9b33-26385f31990a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:17.828114 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.827876    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzcwv\" (UniqueName: \"kubernetes.io/projected/2e5749b0-4658-447f-9b33-26385f31990a-kube-api-access-bzcwv\") pod \"klusterlet-addon-workmgr-69478447dc-2jscf\" (UID: \"2e5749b0-4658-447f-9b33-26385f31990a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:17.828326 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.828300    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-ca\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.828396 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.828355    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e5749b0-4658-447f-9b33-26385f31990a-tmp\") pod \"klusterlet-addon-workmgr-69478447dc-2jscf\" (UID: \"2e5749b0-4658-447f-9b33-26385f31990a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:17.828618 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.828594    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.830423 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.830388    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.830516 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.830438    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-hub\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.830578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.830556    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-ca\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.830613 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.830591    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.830661 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.830645    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2e5749b0-4658-447f-9b33-26385f31990a-klusterlet-config\") pod \"klusterlet-addon-workmgr-69478447dc-2jscf\" (UID: \"2e5749b0-4658-447f-9b33-26385f31990a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:17.836555 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.836535    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzcwv\" (UniqueName: \"kubernetes.io/projected/2e5749b0-4658-447f-9b33-26385f31990a-kube-api-access-bzcwv\") pod \"klusterlet-addon-workmgr-69478447dc-2jscf\" (UID: \"2e5749b0-4658-447f-9b33-26385f31990a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:17.836969 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.836948    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6xf\" (UniqueName: \"kubernetes.io/projected/e2e0ef7e-c9a3-450e-8756-3422c614f6c8-kube-api-access-qm6xf\") pod \"cluster-proxy-proxy-agent-778ff46ccf-wsm7j\" (UID: \"e2e0ef7e-c9a3-450e-8756-3422c614f6c8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:17.914922 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.914871    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:17.951704 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:17.951672    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:05:18.044429 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:05:18.044398    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5749b0_4658_447f_9b33_26385f31990a.slice/crio-d77c653d62ed81b386e3a7aa45de54f3936935b9b6d8b45b8dd7c440007cc156 WatchSource:0}: Error finding container d77c653d62ed81b386e3a7aa45de54f3936935b9b6d8b45b8dd7c440007cc156: Status 404 returned error can't find the container with id d77c653d62ed81b386e3a7aa45de54f3936935b9b6d8b45b8dd7c440007cc156
Jul 09 18:05:18.052577 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:18.052505    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"]
Jul 09 18:05:18.090942 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:18.090854    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"]
Jul 09 18:05:18.093803 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:05:18.093768    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2e0ef7e_c9a3_450e_8756_3422c614f6c8.slice/crio-332df2c734e929ebfed39c38acce7f7432fce34e352a4e51e43cabab30886326 WatchSource:0}: Error finding container 332df2c734e929ebfed39c38acce7f7432fce34e352a4e51e43cabab30886326: Status 404 returned error can't find the container with id 332df2c734e929ebfed39c38acce7f7432fce34e352a4e51e43cabab30886326
Jul 09 18:05:18.491288 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:18.491201    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" event={"ID":"e2e0ef7e-c9a3-450e-8756-3422c614f6c8","Type":"ContainerStarted","Data":"332df2c734e929ebfed39c38acce7f7432fce34e352a4e51e43cabab30886326"}
Jul 09 18:05:18.492361 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:18.492330    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf" event={"ID":"2e5749b0-4658-447f-9b33-26385f31990a","Type":"ContainerStarted","Data":"d77c653d62ed81b386e3a7aa45de54f3936935b9b6d8b45b8dd7c440007cc156"}
Jul 09 18:05:23.507664 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:23.507626    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" event={"ID":"e2e0ef7e-c9a3-450e-8756-3422c614f6c8","Type":"ContainerStarted","Data":"823c922dfb303b543da866f20123ff366ab66fc7337efc6510cab0558281f83e"}
Jul 09 18:05:23.507664 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:23.507670    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" event={"ID":"e2e0ef7e-c9a3-450e-8756-3422c614f6c8","Type":"ContainerStarted","Data":"71ee25357d0beab6cbdc008fa5be3ed7083e8ef4756a14295ab8a0dcd34173fb"}
Jul 09 18:05:23.508154 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:23.507683    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" event={"ID":"e2e0ef7e-c9a3-450e-8756-3422c614f6c8","Type":"ContainerStarted","Data":"dffb6a7285b0b9898bc3b6c233d98f4d31e69abdeeca4e86a292893ede072b28"}
Jul 09 18:05:23.528415 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:23.528361    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" podStartSLOduration=1.882539044 podStartE2EDuration="6.528347695s" totalImagesPullingTime="4.645808651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:23.527882755 +0000 UTC m=+77.861746832" watchObservedRunningTime="2026-07-09 18:05:23.528347695 +0000 UTC m=+77.862211769"
Jul 09 18:05:39.553469 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:39.553381    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf" event={"ID":"2e5749b0-4658-447f-9b33-26385f31990a","Type":"ContainerStarted","Data":"c15b73d210d56c33667bc20c758390edd39371a1b1dbe33fecdd4be500d2f75d"}
Jul 09 18:05:39.553879 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:39.553567    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:39.554989 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:39.554968    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:05:39.570846 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:39.570804    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf" podStartSLOduration=1.334770361 podStartE2EDuration="22.570791781s" totalImagesPullingTime="21.23602142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:39.568971476 +0000 UTC m=+93.902835572" watchObservedRunningTime="2026-07-09 18:05:39.570791781 +0000 UTC m=+93.904655858"
Jul 09 18:05:45.326558 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:45.326511    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:05:45.326871 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:05:45.326670    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:45.326871 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:05:45.326753    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls podName:2b91a8a8-43dc-4f19-b55c-62e2b06801f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:06:49.326734433 +0000 UTC m=+163.660598509 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls") pod "dns-default-46xvq" (UID: "2b91a8a8-43dc-4f19-b55c-62e2b06801f2") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:48.494631 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:05:48.494600    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bth88"
Jul 09 18:06:15.930441 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:06:15.930402    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:06:15.930848 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:06:15.930538    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 18:06:15.930848 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:06:15.930601    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs podName:c4b61885-5a4f-4355-ab6c-f95538b7992b nodeName:}" failed. No retries permitted until 2026-07-09 18:08:17.930582094 +0000 UTC m=+252.264446154 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs") pod "network-metrics-daemon-f84z2" (UID: "c4b61885-5a4f-4355-ab6c-f95538b7992b") : secret "metrics-daemon-secret" not found
Jul 09 18:06:44.455656 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:06:44.455595    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-46xvq" podUID="2b91a8a8-43dc-4f19-b55c-62e2b06801f2"
Jul 09 18:06:44.734877 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:06:44.734798    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46xvq"
Jul 09 18:06:46.240861 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:06:46.240825    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:06:49.364169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:06:49.364127    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:06:49.364535 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:06:49.364295    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:06:49.364535 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:06:49.364388    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls podName:2b91a8a8-43dc-4f19-b55c-62e2b06801f2 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:51.364363893 +0000 UTC m=+285.698227956 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls") pod "dns-default-46xvq" (UID: "2b91a8a8-43dc-4f19-b55c-62e2b06801f2") : secret "dns-default-metrics-tls" not found
Jul 09 18:07:00.222212 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:07:00.222114    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:07:39.554060 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:07:39.554009    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf" podUID="2e5749b0-4658-447f-9b33-26385f31990a" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
Jul 09 18:07:39.890582 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:07:39.890549    2568 generic.go:356] "Generic (PLEG): container finished" podID="2e5749b0-4658-447f-9b33-26385f31990a" containerID="c15b73d210d56c33667bc20c758390edd39371a1b1dbe33fecdd4be500d2f75d" exitCode=1
Jul 09 18:07:39.890737 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:07:39.890618    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf" event={"ID":"2e5749b0-4658-447f-9b33-26385f31990a","Type":"ContainerDied","Data":"c15b73d210d56c33667bc20c758390edd39371a1b1dbe33fecdd4be500d2f75d"}
Jul 09 18:07:39.890959 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:07:39.890946    2568 scope.go:117] "RemoveContainer" containerID="c15b73d210d56c33667bc20c758390edd39371a1b1dbe33fecdd4be500d2f75d"
Jul 09 18:07:40.895094 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:07:40.895061    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf" event={"ID":"2e5749b0-4658-447f-9b33-26385f31990a","Type":"ContainerStarted","Data":"711a7268b18e030f8e2b7431b56c86fa3c82b90063c26f1b0ec08d2cbb900cef"}
Jul 09 18:07:40.895483 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:07:40.895338    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:07:40.895925 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:07:40.895908    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69478447dc-2jscf"
Jul 09 18:08:13.378243 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.378202    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wjp96"]
Jul 09 18:08:13.381439 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.381420    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.385842 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.385820    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:08:13.385969 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.385822    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:08:13.385969 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.385914    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lw2dl\""
Jul 09 18:08:13.387017 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.386999    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:08:13.387125 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.387033    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:08:13.412304 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.412275    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wjp96"]
Jul 09 18:08:13.452979 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.452953    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-data-volume\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.453147 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.453012    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-crio-socket\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.453147 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.453040    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.453147 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.453062    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.453147 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.453088    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzsz7\" (UniqueName: \"kubernetes.io/projected/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-api-access-hzsz7\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.553421 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.553381    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-data-volume\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.553569 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.553440    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-crio-socket\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.553569 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.553456    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.553569 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.553473    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.553569 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.553491    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsz7\" (UniqueName: \"kubernetes.io/projected/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-api-access-hzsz7\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.553743 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:13.553571    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:08:13.553743 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:13.553616    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:13.553743 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.553626    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-crio-socket\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.553743 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:13.553656    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm podName:1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:14.053638156 +0000 UTC m=+248.387502211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wjp96" (UID: "1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3") : configmap "kube-rbac-proxy" not found
Jul 09 18:08:13.553743 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:13.553674    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls podName:1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:14.053665186 +0000 UTC m=+248.387529248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wjp96" (UID: "1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:13.553935 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.553760    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-data-volume\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:13.573936 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:13.573883    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzsz7\" (UniqueName: \"kubernetes.io/projected/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-api-access-hzsz7\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:14.056111 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:14.056071    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:14.056111 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:14.056115    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:14.056369 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:14.056185    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:08:14.056369 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:14.056246    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:14.056369 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:14.056248    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm podName:1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:15.056233584 +0000 UTC m=+249.390097640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wjp96" (UID: "1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3") : configmap "kube-rbac-proxy" not found
Jul 09 18:08:14.056369 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:14.056304    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls podName:1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:15.056288951 +0000 UTC m=+249.390153011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wjp96" (UID: "1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:15.063561 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:15.063505    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:15.063561 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:15.063566    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:15.064059 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:15.063621    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:08:15.064059 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:15.063660    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:15.064059 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:15.063690    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm podName:1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:17.063670855 +0000 UTC m=+251.397534930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wjp96" (UID: "1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3") : configmap "kube-rbac-proxy" not found
Jul 09 18:08:15.064059 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:15.063707    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls podName:1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:17.063699379 +0000 UTC m=+251.397563445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wjp96" (UID: "1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:15.170222 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:15.170192    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kg2kp_24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10/dns-node-resolver/0.log"
Jul 09 18:08:16.371807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:16.371770    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jlzk7_b3bd61c5-6cd4-434e-8b3d-487cbacf897f/node-ca/0.log"
Jul 09 18:08:17.075113 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:17.075072    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:17.075113 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:17.075110    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:17.075321 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:17.075199    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:08:17.075321 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:17.075226    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:17.075321 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:17.075274    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls podName:1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:21.075255782 +0000 UTC m=+255.409119838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wjp96" (UID: "1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:17.075321 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:17.075290    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm podName:1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:21.075281426 +0000 UTC m=+255.409145487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wjp96" (UID: "1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3") : configmap "kube-rbac-proxy" not found
Jul 09 18:08:17.370692 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:17.370619    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gjkgm_1cae8e1a-1640-4639-8783-74995450b5a3/serve-healthcheck-canary/0.log"
Jul 09 18:08:17.980159 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:17.980123    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:08:17.980533 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:17.980278    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 18:08:17.980533 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:17.980343    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs podName:c4b61885-5a4f-4355-ab6c-f95538b7992b nodeName:}" failed. No retries permitted until 2026-07-09 18:10:19.980323809 +0000 UTC m=+374.314187872 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs") pod "network-metrics-daemon-f84z2" (UID: "c4b61885-5a4f-4355-ab6c-f95538b7992b") : secret "metrics-daemon-secret" not found
Jul 09 18:08:21.100576 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:21.100536    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:21.100576 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:21.100574    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:21.101101 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:21.100661    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:08:21.101101 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:21.100708    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:21.101101 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:21.100727    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm podName:1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:29.100712051 +0000 UTC m=+263.434576111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wjp96" (UID: "1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3") : configmap "kube-rbac-proxy" not found
Jul 09 18:08:21.101101 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:21.100747    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls podName:1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:29.100736332 +0000 UTC m=+263.434600388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wjp96" (UID: "1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:29.157853 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:29.157765    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:29.157853 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:29.157807    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:29.158350 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:29.158328    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:29.160117 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:29.160092    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjp96\" (UID: \"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3\") " pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:29.289719 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:29.289680    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wjp96"
Jul 09 18:08:29.412301 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:29.412230    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wjp96"]
Jul 09 18:08:29.415588 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:08:29.415563    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd8fc5a_20f1_4ef9_a4c2_6ac79bc92cd3.slice/crio-5c5d0929d67cb66e4a1cbab8b2fcc97e3378848f1d0224fa4ed66e90e08b9ca4 WatchSource:0}: Error finding container 5c5d0929d67cb66e4a1cbab8b2fcc97e3378848f1d0224fa4ed66e90e08b9ca4: Status 404 returned error can't find the container with id 5c5d0929d67cb66e4a1cbab8b2fcc97e3378848f1d0224fa4ed66e90e08b9ca4
Jul 09 18:08:30.033218 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:30.033182    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wjp96" event={"ID":"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3","Type":"ContainerStarted","Data":"5c5d0929d67cb66e4a1cbab8b2fcc97e3378848f1d0224fa4ed66e90e08b9ca4"}
Jul 09 18:08:34.048364 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:34.048330    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wjp96" event={"ID":"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3","Type":"ContainerStarted","Data":"30050ca4282d5cf003dd9131107b50c3083452eb6e30ce4910da7b596ef96031"}
Jul 09 18:08:34.048730 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:34.048386    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wjp96" event={"ID":"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3","Type":"ContainerStarted","Data":"995907d1c1897d8af40ac0dee2562ccafd953c50af04384a4606650fd86311ac"}
Jul 09 18:08:37.060593 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:37.060554    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wjp96" event={"ID":"1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3","Type":"ContainerStarted","Data":"bde4f8695356bf21c2916d5a5df1b4c696b433c87202991b0719f35fcc8cb3f5"}
Jul 09 18:08:37.077973 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:37.077928    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wjp96" podStartSLOduration=17.235991333 podStartE2EDuration="24.077913859s" totalImagesPullingTime="6.841922526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:08:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:37.077264972 +0000 UTC m=+271.411129077" watchObservedRunningTime="2026-07-09 18:08:37.077913859 +0000 UTC m=+271.411777928"
Jul 09 18:08:47.736062 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:08:47.736018    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-46xvq" podUID="2b91a8a8-43dc-4f19-b55c-62e2b06801f2"
Jul 09 18:08:48.095844 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:48.095813    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46xvq"
Jul 09 18:08:50.900171 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:50.900141    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"]
Jul 09 18:08:51.067861 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.067821    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"]
Jul 09 18:08:51.067861 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.067866    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vhn4h"]
Jul 09 18:08:51.068074 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.067961    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.072517 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.072494    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 09 18:08:51.072719 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.072702    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:08:51.072865 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.072729    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:08:51.072865 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.072745    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 09 18:08:51.072865 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.072755    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:08:51.072865 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.072767    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kcw29\""
Jul 09 18:08:51.073404 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.073386    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 09 18:08:51.093987 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.093967    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.097105 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.097086    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:08:51.097282 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.097264    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xkn2w\""
Jul 09 18:08:51.097350 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.097298    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:08:51.097350 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.097315    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:08:51.117821 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.117797    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.117955 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.117830    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.117955 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.117849    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.117955 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.117868    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz5nv\" (UniqueName: \"kubernetes.io/projected/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-api-access-nz5nv\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.117955 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.117952    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5a686-b844-4583-a2c0-9046a49ccc21-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.118092 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.117981    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b4c5a686-b844-4583-a2c0-9046a49ccc21-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.219096 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219019    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/662d0433-b416-49a5-b4f8-0849b5508cc4-root\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.219096 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219056    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.219096 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219077    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/662d0433-b416-49a5-b4f8-0849b5508cc4-metrics-client-ca\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.219293 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219229    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.219293 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219270    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.219359 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219292    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8mw\" (UniqueName: \"kubernetes.io/projected/662d0433-b416-49a5-b4f8-0849b5508cc4-kube-api-access-nt8mw\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.219359 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219315    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b4c5a686-b844-4583-a2c0-9046a49ccc21-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.219359 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219329    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-wtmp\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.219508 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219454    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-textfile\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.219508 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219490    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/662d0433-b416-49a5-b4f8-0849b5508cc4-sys\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.219606 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219527    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.219606 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219586    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.219707 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219630    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b4c5a686-b844-4583-a2c0-9046a49ccc21-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.219707 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219639    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz5nv\" (UniqueName: \"kubernetes.io/projected/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-api-access-nz5nv\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.219707 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219673    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5a686-b844-4583-a2c0-9046a49ccc21-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.219707 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.219700    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-tls\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.220165 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.220144    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.220377 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.220356    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5a686-b844-4583-a2c0-9046a49ccc21-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.221479 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.221462    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.221695 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.221678    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.227788 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.227769    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz5nv\" (UniqueName: \"kubernetes.io/projected/b4c5a686-b844-4583-a2c0-9046a49ccc21-kube-api-access-nz5nv\") pod \"kube-state-metrics-785778ff74-m4rhb\" (UID: \"b4c5a686-b844-4583-a2c0-9046a49ccc21\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.320483 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320447    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/662d0433-b416-49a5-b4f8-0849b5508cc4-metrics-client-ca\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.320483 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320486    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.320749 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320516    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8mw\" (UniqueName: \"kubernetes.io/projected/662d0433-b416-49a5-b4f8-0849b5508cc4-kube-api-access-nt8mw\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.320749 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320572    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-wtmp\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.320749 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320649    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-textfile\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.320749 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320675    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/662d0433-b416-49a5-b4f8-0849b5508cc4-sys\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.320749 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320711    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.321023 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320749    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-wtmp\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.321023 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320756    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-tls\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.321023 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320823    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/662d0433-b416-49a5-b4f8-0849b5508cc4-root\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.321023 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320917    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/662d0433-b416-49a5-b4f8-0849b5508cc4-root\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.321023 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.321018    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-textfile\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.321256 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.320788    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/662d0433-b416-49a5-b4f8-0849b5508cc4-sys\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.321256 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.321162    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.321256 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.321189    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/662d0433-b416-49a5-b4f8-0849b5508cc4-metrics-client-ca\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.322940 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.322919    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.323043 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.323025    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/662d0433-b416-49a5-b4f8-0849b5508cc4-node-exporter-tls\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.331208 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.331183    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8mw\" (UniqueName: \"kubernetes.io/projected/662d0433-b416-49a5-b4f8-0849b5508cc4-kube-api-access-nt8mw\") pod \"node-exporter-vhn4h\" (UID: \"662d0433-b416-49a5-b4f8-0849b5508cc4\") " pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.377216 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.377183    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"
Jul 09 18:08:51.402616 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.402581    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vhn4h"
Jul 09 18:08:51.421843 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.421816    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:08:51.424802 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.424777    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b91a8a8-43dc-4f19-b55c-62e2b06801f2-metrics-tls\") pod \"dns-default-46xvq\" (UID: \"2b91a8a8-43dc-4f19-b55c-62e2b06801f2\") " pod="openshift-dns/dns-default-46xvq"
Jul 09 18:08:51.436679 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:08:51.436645    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662d0433_b416_49a5_b4f8_0849b5508cc4.slice/crio-54a9295c7e692fd0298b4f1ac5a01d97c0783a748e14314b1124f439b03dac7a WatchSource:0}: Error finding container 54a9295c7e692fd0298b4f1ac5a01d97c0783a748e14314b1124f439b03dac7a: Status 404 returned error can't find the container with id 54a9295c7e692fd0298b4f1ac5a01d97c0783a748e14314b1124f439b03dac7a
Jul 09 18:08:51.532405 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.532374    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-m4rhb"]
Jul 09 18:08:51.535032 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:08:51.535007    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c5a686_b844_4583_a2c0_9046a49ccc21.slice/crio-af6bc5755bc43b3f1c8ac9f1b74a8ff3b69ce64e1ad9ea778c63ee5f4d66bdd8 WatchSource:0}: Error finding container af6bc5755bc43b3f1c8ac9f1b74a8ff3b69ce64e1ad9ea778c63ee5f4d66bdd8: Status 404 returned error can't find the container with id af6bc5755bc43b3f1c8ac9f1b74a8ff3b69ce64e1ad9ea778c63ee5f4d66bdd8
Jul 09 18:08:51.699568 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.699540    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jd75b\""
Jul 09 18:08:51.706835 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.706810    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46xvq"
Jul 09 18:08:51.866200 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:51.866172    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-46xvq"]
Jul 09 18:08:51.868936 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:08:51.868907    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b91a8a8_43dc_4f19_b55c_62e2b06801f2.slice/crio-775233e3eee68f74e779edeb59a43aaf9f194d8e6ff9e18134bf19eaf46a229a WatchSource:0}: Error finding container 775233e3eee68f74e779edeb59a43aaf9f194d8e6ff9e18134bf19eaf46a229a: Status 404 returned error can't find the container with id 775233e3eee68f74e779edeb59a43aaf9f194d8e6ff9e18134bf19eaf46a229a
Jul 09 18:08:52.108171 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:52.108129    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46xvq" event={"ID":"2b91a8a8-43dc-4f19-b55c-62e2b06801f2","Type":"ContainerStarted","Data":"775233e3eee68f74e779edeb59a43aaf9f194d8e6ff9e18134bf19eaf46a229a"}
Jul 09 18:08:52.109471 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:52.109445    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb" event={"ID":"b4c5a686-b844-4583-a2c0-9046a49ccc21","Type":"ContainerStarted","Data":"af6bc5755bc43b3f1c8ac9f1b74a8ff3b69ce64e1ad9ea778c63ee5f4d66bdd8"}
Jul 09 18:08:52.110782 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:52.110752    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhn4h" event={"ID":"662d0433-b416-49a5-b4f8-0849b5508cc4","Type":"ContainerStarted","Data":"54a9295c7e692fd0298b4f1ac5a01d97c0783a748e14314b1124f439b03dac7a"}
Jul 09 18:08:53.123330 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:53.123289    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb" event={"ID":"b4c5a686-b844-4583-a2c0-9046a49ccc21","Type":"ContainerStarted","Data":"559a5963a1f340cbd9d1a1dc333f31c52454f2cbd99249ba26fa30b0445b9760"}
Jul 09 18:08:53.123719 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:53.123409    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb" event={"ID":"b4c5a686-b844-4583-a2c0-9046a49ccc21","Type":"ContainerStarted","Data":"7b4ba9e187cf0b30037e6c37abdcbdd372f0ddf69108c0f15e242842fbaf364f"}
Jul 09 18:08:53.129553 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:53.129525    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhn4h" event={"ID":"662d0433-b416-49a5-b4f8-0849b5508cc4","Type":"ContainerStarted","Data":"aaad3a03e4193e485d5797b932a43e80abf0754552d2083b2dcb042ef05ac5a6"}
Jul 09 18:08:54.135821 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:54.135784    2568 generic.go:356] "Generic (PLEG): container finished" podID="662d0433-b416-49a5-b4f8-0849b5508cc4" containerID="aaad3a03e4193e485d5797b932a43e80abf0754552d2083b2dcb042ef05ac5a6" exitCode=0
Jul 09 18:08:54.135821 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:54.135817    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhn4h" event={"ID":"662d0433-b416-49a5-b4f8-0849b5508cc4","Type":"ContainerDied","Data":"aaad3a03e4193e485d5797b932a43e80abf0754552d2083b2dcb042ef05ac5a6"}
Jul 09 18:08:54.137904 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:54.137869    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46xvq" event={"ID":"2b91a8a8-43dc-4f19-b55c-62e2b06801f2","Type":"ContainerStarted","Data":"c467758e3402792728fbd5c781fa2e5f60cfc7a2f473ccc70ab8e7bbede69115"}
Jul 09 18:08:54.138028 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:54.137929    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46xvq" event={"ID":"2b91a8a8-43dc-4f19-b55c-62e2b06801f2","Type":"ContainerStarted","Data":"c46e5245e1b8f823d34bc7e2c504755fcb89346c74ec8da08ad5815fa822f708"}
Jul 09 18:08:54.138028 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:54.137988    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-46xvq"
Jul 09 18:08:54.140584 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:54.140559    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb" event={"ID":"b4c5a686-b844-4583-a2c0-9046a49ccc21","Type":"ContainerStarted","Data":"cb5c8a4ff59d5b034c791a9fbee15415d3c5c502744d380a18d58b4ea5ae7fe6"}
Jul 09 18:08:54.173430 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:54.173378    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-46xvq" podStartSLOduration=251.097898486 podStartE2EDuration="4m13.173360761s" totalImagesPullingTime="2.075462275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:54.172511109 +0000 UTC m=+288.506375207" watchObservedRunningTime="2026-07-09 18:08:54.173360761 +0000 UTC m=+288.507224838"
Jul 09 18:08:54.191129 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:54.191082    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-m4rhb" podStartSLOduration=2.760292765 podStartE2EDuration="4.191065407s" totalImagesPullingTime="1.430772642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:54.190400496 +0000 UTC m=+288.524264574" watchObservedRunningTime="2026-07-09 18:08:54.191065407 +0000 UTC m=+288.524929483"
Jul 09 18:08:55.146269 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:55.146232    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhn4h" event={"ID":"662d0433-b416-49a5-b4f8-0849b5508cc4","Type":"ContainerStarted","Data":"cac93c3a95e5b9d896b63a1e6528dc55cba7ec138558c22af05374fa37a7ebe2"}
Jul 09 18:08:55.146648 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:08:55.146300    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhn4h" event={"ID":"662d0433-b416-49a5-b4f8-0849b5508cc4","Type":"ContainerStarted","Data":"3ae4bd482207ad0e3b0abb1bec8b8adc45b62388472c7258e02034ea64b1a016"}
Jul 09 18:09:03.222690 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:09:03.222653    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-f84z2" podUID="c4b61885-5a4f-4355-ab6c-f95538b7992b"
Jul 09 18:09:04.149534 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:04.149503    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-46xvq"
Jul 09 18:09:04.169084 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:04.169012    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vhn4h" podStartSLOduration=12.554963342 podStartE2EDuration="14.168991197s" totalImagesPullingTime="1.526964149s" totalInitContainerRuntime="87.063706ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:55.193132035 +0000 UTC m=+289.526996113" watchObservedRunningTime="2026-07-09 18:09:04.168991197 +0000 UTC m=+298.502855276"
Jul 09 18:09:06.100616 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:06.100590    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:09:06.101591 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:06.101569    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:09:06.104828 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:06.104805    2568 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:09:15.221660 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:15.221631    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:09:17.953703 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:17.953665    2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" podUID="e2e0ef7e-c9a3-450e-8756-3422c614f6c8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 18:09:25.886176 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.886142    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b58c9d8df-fl2m4"]
Jul 09 18:09:25.888548 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.888530    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:25.891311 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.891289    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 18:09:25.891427 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.891335    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 18:09:25.892608 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.892581    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 18:09:25.892717 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.892622    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 18:09:25.892717 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.892580    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 18:09:25.892818 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.892790    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 18:09:25.892818 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.892792    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 18:09:25.892951 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.892823    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7bkfr\""
Jul 09 18:09:25.897383 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.897362    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 18:09:25.899105 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.899085    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b58c9d8df-fl2m4"]
Jul 09 18:09:25.999275 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.999238    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-oauth-config\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:25.999449 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.999287    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-console-config\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:25.999449 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.999310    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w7rv\" (UniqueName: \"kubernetes.io/projected/1560665c-28c8-4777-bdce-76e970c9a3bf-kube-api-access-2w7rv\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:25.999449 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.999369    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-oauth-serving-cert\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:25.999449 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.999411    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-trusted-ca-bundle\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:25.999449 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.999440    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-serving-cert\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:25.999606 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:25.999470    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-service-ca\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.100165 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.100127    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-service-ca\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.100335 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.100184    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-oauth-config\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.100335 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.100214    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-console-config\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.100335 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.100231    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w7rv\" (UniqueName: \"kubernetes.io/projected/1560665c-28c8-4777-bdce-76e970c9a3bf-kube-api-access-2w7rv\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.100335 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.100250    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-oauth-serving-cert\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.100335 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.100269    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-trusted-ca-bundle\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.100335 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.100290    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-serving-cert\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.101019 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.100992    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-oauth-serving-cert\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.101151 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.100994    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-service-ca\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.101151 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.101054    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-console-config\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.101151 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.101121    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-trusted-ca-bundle\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.102873 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.102854    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-oauth-config\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.102988 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.102860    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-serving-cert\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.108826 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.108800    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w7rv\" (UniqueName: \"kubernetes.io/projected/1560665c-28c8-4777-bdce-76e970c9a3bf-kube-api-access-2w7rv\") pod \"console-b58c9d8df-fl2m4\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") " pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.198666 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.198565    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:26.332733 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.332700    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b58c9d8df-fl2m4"]
Jul 09 18:09:26.336323 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:09:26.336296    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1560665c_28c8_4777_bdce_76e970c9a3bf.slice/crio-f63a8f6529723f84a0d42a238299ba0c3955039416e3ee948395941f1a504b2c WatchSource:0}: Error finding container f63a8f6529723f84a0d42a238299ba0c3955039416e3ee948395941f1a504b2c: Status 404 returned error can't find the container with id f63a8f6529723f84a0d42a238299ba0c3955039416e3ee948395941f1a504b2c
Jul 09 18:09:26.338195 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:26.338180    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:09:27.252535 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:27.252491    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b58c9d8df-fl2m4" event={"ID":"1560665c-28c8-4777-bdce-76e970c9a3bf","Type":"ContainerStarted","Data":"f63a8f6529723f84a0d42a238299ba0c3955039416e3ee948395941f1a504b2c"}
Jul 09 18:09:27.953430 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:27.953382    2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" podUID="e2e0ef7e-c9a3-450e-8756-3422c614f6c8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 18:09:29.260516 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:29.260479    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b58c9d8df-fl2m4" event={"ID":"1560665c-28c8-4777-bdce-76e970c9a3bf","Type":"ContainerStarted","Data":"c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996"}
Jul 09 18:09:29.278351 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:29.278287    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b58c9d8df-fl2m4" podStartSLOduration=1.801149196 podStartE2EDuration="4.278271231s" totalImagesPullingTime="2.477122035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:29.277244865 +0000 UTC m=+323.611108958" watchObservedRunningTime="2026-07-09 18:09:29.278271231 +0000 UTC m=+323.612135308"
Jul 09 18:09:36.199676 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:36.199638    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:36.200070 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:36.199727    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:36.204254 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:36.204236    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:36.285922 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:36.285880    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:09:36.673039 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:36.673011    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-m4rhb_b4c5a686-b844-4583-a2c0-9046a49ccc21/kube-state-metrics/0.log"
Jul 09 18:09:36.873114 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:36.873062    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-m4rhb_b4c5a686-b844-4583-a2c0-9046a49ccc21/kube-rbac-proxy-main/0.log"
Jul 09 18:09:37.074967 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:37.074921    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-m4rhb_b4c5a686-b844-4583-a2c0-9046a49ccc21/kube-rbac-proxy-self/0.log"
Jul 09 18:09:37.953461 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:37.953424    2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" podUID="e2e0ef7e-c9a3-450e-8756-3422c614f6c8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 18:09:37.953927 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:37.953499    2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j"
Jul 09 18:09:37.954018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:37.953977    2568 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"823c922dfb303b543da866f20123ff366ab66fc7337efc6510cab0558281f83e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 09 18:09:37.954065 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:37.954037    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" podUID="e2e0ef7e-c9a3-450e-8756-3422c614f6c8" containerName="service-proxy" containerID="cri-o://823c922dfb303b543da866f20123ff366ab66fc7337efc6510cab0558281f83e" gracePeriod=30
Jul 09 18:09:38.272752 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:38.272727    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhn4h_662d0433-b416-49a5-b4f8-0849b5508cc4/init-textfile/0.log"
Jul 09 18:09:38.289557 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:38.289530    2568 generic.go:356] "Generic (PLEG): container finished" podID="e2e0ef7e-c9a3-450e-8756-3422c614f6c8" containerID="823c922dfb303b543da866f20123ff366ab66fc7337efc6510cab0558281f83e" exitCode=2
Jul 09 18:09:38.289708 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:38.289604    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" event={"ID":"e2e0ef7e-c9a3-450e-8756-3422c614f6c8","Type":"ContainerDied","Data":"823c922dfb303b543da866f20123ff366ab66fc7337efc6510cab0558281f83e"}
Jul 09 18:09:38.289708 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:38.289650    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778ff46ccf-wsm7j" event={"ID":"e2e0ef7e-c9a3-450e-8756-3422c614f6c8","Type":"ContainerStarted","Data":"c57eb8a1b0f9c0ca8a84e29861d2da0876d46fcae0aac25e52d85916a68418c6"}
Jul 09 18:09:38.473815 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:38.473786    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhn4h_662d0433-b416-49a5-b4f8-0849b5508cc4/node-exporter/0.log"
Jul 09 18:09:38.672539 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:38.672432    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhn4h_662d0433-b416-49a5-b4f8-0849b5508cc4/kube-rbac-proxy/0.log"
Jul 09 18:09:44.674285 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:44.674256    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b58c9d8df-fl2m4_1560665c-28c8-4777-bdce-76e970c9a3bf/console/0.log"
Jul 09 18:09:45.673358 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:09:45.673288    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gjkgm_1cae8e1a-1640-4639-8783-74995450b5a3/serve-healthcheck-canary/0.log"
Jul 09 18:10:19.790512 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:19.790477    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b58c9d8df-fl2m4"]
Jul 09 18:10:20.013862 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:20.013826    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:10:20.016181 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:20.016147    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4b61885-5a4f-4355-ab6c-f95538b7992b-metrics-certs\") pod \"network-metrics-daemon-f84z2\" (UID: \"c4b61885-5a4f-4355-ab6c-f95538b7992b\") " pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:10:20.025012 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:20.024990    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7k8ns\""
Jul 09 18:10:20.032405 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:20.032380    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f84z2"
Jul 09 18:10:20.157771 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:20.157741    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f84z2"]
Jul 09 18:10:20.161582 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:10:20.161549    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b61885_5a4f_4355_ab6c_f95538b7992b.slice/crio-72b2409ff33bf712bf4bfddff79d2cf8e0bcfeadc878622710321d63ddf1c5f7 WatchSource:0}: Error finding container 72b2409ff33bf712bf4bfddff79d2cf8e0bcfeadc878622710321d63ddf1c5f7: Status 404 returned error can't find the container with id 72b2409ff33bf712bf4bfddff79d2cf8e0bcfeadc878622710321d63ddf1c5f7
Jul 09 18:10:20.429559 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:20.429472    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f84z2" event={"ID":"c4b61885-5a4f-4355-ab6c-f95538b7992b","Type":"ContainerStarted","Data":"72b2409ff33bf712bf4bfddff79d2cf8e0bcfeadc878622710321d63ddf1c5f7"}
Jul 09 18:10:21.434440 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:21.434354    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f84z2" event={"ID":"c4b61885-5a4f-4355-ab6c-f95538b7992b","Type":"ContainerStarted","Data":"ed0629fc51719ecd33a297ac88797a98c439d9b13b752526c6e96805954d5b1b"}
Jul 09 18:10:21.434440 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:21.434411    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f84z2" event={"ID":"c4b61885-5a4f-4355-ab6c-f95538b7992b","Type":"ContainerStarted","Data":"03b3f851a82507f693bf299e621254b8a08fa318fdba68e55c8750a8704bc0dc"}
Jul 09 18:10:21.451421 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:21.451374    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f84z2" podStartSLOduration=374.433224564 podStartE2EDuration="6m15.451358339s" totalImagesPullingTime="1.018133775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:21.449841896 +0000 UTC m=+375.783705987" watchObservedRunningTime="2026-07-09 18:10:21.451358339 +0000 UTC m=+375.785222449"
Jul 09 18:10:44.810022 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:44.809979    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b58c9d8df-fl2m4" podUID="1560665c-28c8-4777-bdce-76e970c9a3bf" containerName="console" containerID="cri-o://c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996" gracePeriod=15
Jul 09 18:10:45.057284 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.057260    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:10:45.093406 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.093334    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-oauth-config\") pod \"1560665c-28c8-4777-bdce-76e970c9a3bf\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") "
Jul 09 18:10:45.093406 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.093366    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w7rv\" (UniqueName: \"kubernetes.io/projected/1560665c-28c8-4777-bdce-76e970c9a3bf-kube-api-access-2w7rv\") pod \"1560665c-28c8-4777-bdce-76e970c9a3bf\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") "
Jul 09 18:10:45.093406 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.093387    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-serving-cert\") pod \"1560665c-28c8-4777-bdce-76e970c9a3bf\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") "
Jul 09 18:10:45.093626 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.093410    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-trusted-ca-bundle\") pod \"1560665c-28c8-4777-bdce-76e970c9a3bf\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") "
Jul 09 18:10:45.093626 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.093454    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-oauth-serving-cert\") pod \"1560665c-28c8-4777-bdce-76e970c9a3bf\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") "
Jul 09 18:10:45.093626 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.093501    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-service-ca\") pod \"1560665c-28c8-4777-bdce-76e970c9a3bf\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") "
Jul 09 18:10:45.093771 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.093653    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-console-config\") pod \"1560665c-28c8-4777-bdce-76e970c9a3bf\" (UID: \"1560665c-28c8-4777-bdce-76e970c9a3bf\") "
Jul 09 18:10:45.093933 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.093864    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-service-ca" (OuterVolumeSpecName: "service-ca") pod "1560665c-28c8-4777-bdce-76e970c9a3bf" (UID: "1560665c-28c8-4777-bdce-76e970c9a3bf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:45.093933 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.093922    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1560665c-28c8-4777-bdce-76e970c9a3bf" (UID: "1560665c-28c8-4777-bdce-76e970c9a3bf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:45.094182 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.093964    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1560665c-28c8-4777-bdce-76e970c9a3bf" (UID: "1560665c-28c8-4777-bdce-76e970c9a3bf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:45.094245 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.094178    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-console-config" (OuterVolumeSpecName: "console-config") pod "1560665c-28c8-4777-bdce-76e970c9a3bf" (UID: "1560665c-28c8-4777-bdce-76e970c9a3bf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:45.095627 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.095600    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1560665c-28c8-4777-bdce-76e970c9a3bf-kube-api-access-2w7rv" (OuterVolumeSpecName: "kube-api-access-2w7rv") pod "1560665c-28c8-4777-bdce-76e970c9a3bf" (UID: "1560665c-28c8-4777-bdce-76e970c9a3bf"). InnerVolumeSpecName "kube-api-access-2w7rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:10:45.095716 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.095633    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1560665c-28c8-4777-bdce-76e970c9a3bf" (UID: "1560665c-28c8-4777-bdce-76e970c9a3bf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:45.095716 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.095648    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1560665c-28c8-4777-bdce-76e970c9a3bf" (UID: "1560665c-28c8-4777-bdce-76e970c9a3bf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:45.195115 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.195089    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-trusted-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:10:45.195115 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.195114    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-oauth-serving-cert\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:10:45.195281 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.195123    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-service-ca\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:10:45.195281 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.195132    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1560665c-28c8-4777-bdce-76e970c9a3bf-console-config\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:10:45.195281 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.195140    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-oauth-config\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:10:45.195281 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.195149    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w7rv\" (UniqueName: \"kubernetes.io/projected/1560665c-28c8-4777-bdce-76e970c9a3bf-kube-api-access-2w7rv\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:10:45.195281 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.195158    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1560665c-28c8-4777-bdce-76e970c9a3bf-console-serving-cert\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:10:45.517095 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.517063    2568 generic.go:356] "Generic (PLEG): container finished" podID="1560665c-28c8-4777-bdce-76e970c9a3bf" containerID="c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996" exitCode=0
Jul 09 18:10:45.517267 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.517112    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b58c9d8df-fl2m4"
Jul 09 18:10:45.517267 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.517148    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b58c9d8df-fl2m4" event={"ID":"1560665c-28c8-4777-bdce-76e970c9a3bf","Type":"ContainerDied","Data":"c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996"}
Jul 09 18:10:45.517267 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.517209    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b58c9d8df-fl2m4" event={"ID":"1560665c-28c8-4777-bdce-76e970c9a3bf","Type":"ContainerDied","Data":"f63a8f6529723f84a0d42a238299ba0c3955039416e3ee948395941f1a504b2c"}
Jul 09 18:10:45.517267 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.517249    2568 scope.go:117] "RemoveContainer" containerID="c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996"
Jul 09 18:10:45.530115 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.530066    2568 scope.go:117] "RemoveContainer" containerID="c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996"
Jul 09 18:10:45.530318 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:10:45.530295    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996\": container with ID starting with c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996 not found: ID does not exist" containerID="c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996"
Jul 09 18:10:45.530400 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.530324    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996"} err="failed to get container status \"c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996\": rpc error: code = NotFound desc = could not find container \"c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996\": container with ID starting with c5f9b15c683ddddf686e36a5c2ae93960d8e794903bba9eee5480c9b520ce996 not found: ID does not exist"
Jul 09 18:10:45.541946 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.541920    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b58c9d8df-fl2m4"]
Jul 09 18:10:45.543473 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:45.543450    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b58c9d8df-fl2m4"]
Jul 09 18:10:46.225699 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:10:46.225666    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1560665c-28c8-4777-bdce-76e970c9a3bf" path="/var/lib/kubelet/pods/1560665c-28c8-4777-bdce-76e970c9a3bf/volumes"
Jul 09 18:11:50.168092 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.168018    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"]
Jul 09 18:11:50.168479 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.168243    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1560665c-28c8-4777-bdce-76e970c9a3bf" containerName="console"
Jul 09 18:11:50.168479 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.168254    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1560665c-28c8-4777-bdce-76e970c9a3bf" containerName="console"
Jul 09 18:11:50.168479 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.168291    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1560665c-28c8-4777-bdce-76e970c9a3bf" containerName="console"
Jul 09 18:11:50.170896 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.170870    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:50.173589 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.173558    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 09 18:11:50.173589 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.173585    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 09 18:11:50.173902 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.173869    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 09 18:11:50.173997 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.173914    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 09 18:11:50.174189 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.174173    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 09 18:11:50.176078 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.176055    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-szds9\""
Jul 09 18:11:50.180044 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.180024    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"]
Jul 09 18:11:50.237994 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.237960    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/db150aa9-e7bd-470e-80c8-5d7fbedce34f-certificates\") pod \"keda-metrics-apiserver-559f75f947-9wnsp\" (UID: \"db150aa9-e7bd-470e-80c8-5d7fbedce34f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:50.237994 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.238004    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96plf\" (UniqueName: \"kubernetes.io/projected/db150aa9-e7bd-470e-80c8-5d7fbedce34f-kube-api-access-96plf\") pod \"keda-metrics-apiserver-559f75f947-9wnsp\" (UID: \"db150aa9-e7bd-470e-80c8-5d7fbedce34f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:50.238217 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.238041    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/db150aa9-e7bd-470e-80c8-5d7fbedce34f-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-9wnsp\" (UID: \"db150aa9-e7bd-470e-80c8-5d7fbedce34f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:50.339055 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.339026    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/db150aa9-e7bd-470e-80c8-5d7fbedce34f-certificates\") pod \"keda-metrics-apiserver-559f75f947-9wnsp\" (UID: \"db150aa9-e7bd-470e-80c8-5d7fbedce34f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:50.339055 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.339058    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96plf\" (UniqueName: \"kubernetes.io/projected/db150aa9-e7bd-470e-80c8-5d7fbedce34f-kube-api-access-96plf\") pod \"keda-metrics-apiserver-559f75f947-9wnsp\" (UID: \"db150aa9-e7bd-470e-80c8-5d7fbedce34f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:50.339305 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.339086    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/db150aa9-e7bd-470e-80c8-5d7fbedce34f-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-9wnsp\" (UID: \"db150aa9-e7bd-470e-80c8-5d7fbedce34f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:50.339305 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:11:50.339165    2568 secret.go:281] references non-existent secret key: tls.crt
Jul 09 18:11:50.339305 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:11:50.339185    2568 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 09 18:11:50.339305 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:11:50.339203    2568 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp: references non-existent secret key: tls.crt
Jul 09 18:11:50.339305 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:11:50.339253    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db150aa9-e7bd-470e-80c8-5d7fbedce34f-certificates podName:db150aa9-e7bd-470e-80c8-5d7fbedce34f nodeName:}" failed. No retries permitted until 2026-07-09 18:11:50.839239446 +0000 UTC m=+465.173103502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/db150aa9-e7bd-470e-80c8-5d7fbedce34f-certificates") pod "keda-metrics-apiserver-559f75f947-9wnsp" (UID: "db150aa9-e7bd-470e-80c8-5d7fbedce34f") : references non-existent secret key: tls.crt
Jul 09 18:11:50.339539 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.339404    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/db150aa9-e7bd-470e-80c8-5d7fbedce34f-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-9wnsp\" (UID: \"db150aa9-e7bd-470e-80c8-5d7fbedce34f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:50.348192 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.348169    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96plf\" (UniqueName: \"kubernetes.io/projected/db150aa9-e7bd-470e-80c8-5d7fbedce34f-kube-api-access-96plf\") pod \"keda-metrics-apiserver-559f75f947-9wnsp\" (UID: \"db150aa9-e7bd-470e-80c8-5d7fbedce34f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:50.842904 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.842849    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/db150aa9-e7bd-470e-80c8-5d7fbedce34f-certificates\") pod \"keda-metrics-apiserver-559f75f947-9wnsp\" (UID: \"db150aa9-e7bd-470e-80c8-5d7fbedce34f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:50.845340 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:50.845312    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/db150aa9-e7bd-470e-80c8-5d7fbedce34f-certificates\") pod \"keda-metrics-apiserver-559f75f947-9wnsp\" (UID: \"db150aa9-e7bd-470e-80c8-5d7fbedce34f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:51.081381 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:51.081347    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:51.203084 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:51.203058    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"]
Jul 09 18:11:51.206275 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:11:51.206243    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb150aa9_e7bd_470e_80c8_5d7fbedce34f.slice/crio-e49bbc8991247cf4bc90fffc04c95c6b24843d8523ac33791680277eda3a0f48 WatchSource:0}: Error finding container e49bbc8991247cf4bc90fffc04c95c6b24843d8523ac33791680277eda3a0f48: Status 404 returned error can't find the container with id e49bbc8991247cf4bc90fffc04c95c6b24843d8523ac33791680277eda3a0f48
Jul 09 18:11:51.732144 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:51.732109    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp" event={"ID":"db150aa9-e7bd-470e-80c8-5d7fbedce34f","Type":"ContainerStarted","Data":"e49bbc8991247cf4bc90fffc04c95c6b24843d8523ac33791680277eda3a0f48"}
Jul 09 18:11:54.744104 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:54.744064    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp" event={"ID":"db150aa9-e7bd-470e-80c8-5d7fbedce34f","Type":"ContainerStarted","Data":"7e64b162f48f22ff842399e0625ca292f0abd52eaac27a39e9ab2d7207ab52aa"}
Jul 09 18:11:54.744527 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:54.744249    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:11:54.760819 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:11:54.760763    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp" podStartSLOduration=1.321413243 podStartE2EDuration="4.760746322s" totalImagesPullingTime="3.439333079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:54.760608819 +0000 UTC m=+469.094472896" watchObservedRunningTime="2026-07-09 18:11:54.760746322 +0000 UTC m=+469.094610400"
Jul 09 18:12:05.751984 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:05.751952    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-9wnsp"
Jul 09 18:12:57.287255 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.287222    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-fr588"]
Jul 09 18:12:57.289944 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.289926    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:12:57.293144 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.293118    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 18:12:57.293254 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.293173    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 18:12:57.294648 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.294630    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qslkv\""
Jul 09 18:12:57.294696 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.294630    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 09 18:12:57.303255 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.303231    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-fr588"]
Jul 09 18:12:57.304176 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.304157    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-j4v5l"]
Jul 09 18:12:57.306720 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.306702    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-j4v5l"
Jul 09 18:12:57.309372 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.309351    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 09 18:12:57.309608 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.309587    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-z578d\""
Jul 09 18:12:57.323620 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.323600    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-j4v5l"]
Jul 09 18:12:57.396315 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.396289    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8e5590f-b1e4-4b7f-a109-571830c28dfe-cert\") pod \"kserve-controller-manager-748bdf6488-fr588\" (UID: \"f8e5590f-b1e4-4b7f-a109-571830c28dfe\") " pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:12:57.396424 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.396337    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxmp2\" (UniqueName: \"kubernetes.io/projected/f8e5590f-b1e4-4b7f-a109-571830c28dfe-kube-api-access-sxmp2\") pod \"kserve-controller-manager-748bdf6488-fr588\" (UID: \"f8e5590f-b1e4-4b7f-a109-571830c28dfe\") " pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:12:57.396424 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.396355    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49a22f36-fe11-4c4c-aa8d-19af1a508da2-cert\") pod \"llmisvc-controller-manager-654c557856-j4v5l\" (UID: \"49a22f36-fe11-4c4c-aa8d-19af1a508da2\") " pod="kserve/llmisvc-controller-manager-654c557856-j4v5l"
Jul 09 18:12:57.396930 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.396407    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtgh\" (UniqueName: \"kubernetes.io/projected/49a22f36-fe11-4c4c-aa8d-19af1a508da2-kube-api-access-zrtgh\") pod \"llmisvc-controller-manager-654c557856-j4v5l\" (UID: \"49a22f36-fe11-4c4c-aa8d-19af1a508da2\") " pod="kserve/llmisvc-controller-manager-654c557856-j4v5l"
Jul 09 18:12:57.498326 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.498283    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmp2\" (UniqueName: \"kubernetes.io/projected/f8e5590f-b1e4-4b7f-a109-571830c28dfe-kube-api-access-sxmp2\") pod \"kserve-controller-manager-748bdf6488-fr588\" (UID: \"f8e5590f-b1e4-4b7f-a109-571830c28dfe\") " pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:12:57.498326 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.498328    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49a22f36-fe11-4c4c-aa8d-19af1a508da2-cert\") pod \"llmisvc-controller-manager-654c557856-j4v5l\" (UID: \"49a22f36-fe11-4c4c-aa8d-19af1a508da2\") " pod="kserve/llmisvc-controller-manager-654c557856-j4v5l"
Jul 09 18:12:57.498551 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.498351    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtgh\" (UniqueName: \"kubernetes.io/projected/49a22f36-fe11-4c4c-aa8d-19af1a508da2-kube-api-access-zrtgh\") pod \"llmisvc-controller-manager-654c557856-j4v5l\" (UID: \"49a22f36-fe11-4c4c-aa8d-19af1a508da2\") " pod="kserve/llmisvc-controller-manager-654c557856-j4v5l"
Jul 09 18:12:57.498551 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.498387    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8e5590f-b1e4-4b7f-a109-571830c28dfe-cert\") pod \"kserve-controller-manager-748bdf6488-fr588\" (UID: \"f8e5590f-b1e4-4b7f-a109-571830c28dfe\") " pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:12:57.498551 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:12:57.498450    2568 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 09 18:12:57.498551 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:12:57.498483    2568 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 09 18:12:57.498551 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:12:57.498513    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49a22f36-fe11-4c4c-aa8d-19af1a508da2-cert podName:49a22f36-fe11-4c4c-aa8d-19af1a508da2 nodeName:}" failed. No retries permitted until 2026-07-09 18:12:57.998494418 +0000 UTC m=+532.332358477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/49a22f36-fe11-4c4c-aa8d-19af1a508da2-cert") pod "llmisvc-controller-manager-654c557856-j4v5l" (UID: "49a22f36-fe11-4c4c-aa8d-19af1a508da2") : secret "llmisvc-webhook-server-cert" not found
Jul 09 18:12:57.498551 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:12:57.498543    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e5590f-b1e4-4b7f-a109-571830c28dfe-cert podName:f8e5590f-b1e4-4b7f-a109-571830c28dfe nodeName:}" failed. No retries permitted until 2026-07-09 18:12:57.998526449 +0000 UTC m=+532.332390505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f8e5590f-b1e4-4b7f-a109-571830c28dfe-cert") pod "kserve-controller-manager-748bdf6488-fr588" (UID: "f8e5590f-b1e4-4b7f-a109-571830c28dfe") : secret "kserve-webhook-server-cert" not found
Jul 09 18:12:57.507819 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.507794    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmp2\" (UniqueName: \"kubernetes.io/projected/f8e5590f-b1e4-4b7f-a109-571830c28dfe-kube-api-access-sxmp2\") pod \"kserve-controller-manager-748bdf6488-fr588\" (UID: \"f8e5590f-b1e4-4b7f-a109-571830c28dfe\") " pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:12:57.507951 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:57.507801    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtgh\" (UniqueName: \"kubernetes.io/projected/49a22f36-fe11-4c4c-aa8d-19af1a508da2-kube-api-access-zrtgh\") pod \"llmisvc-controller-manager-654c557856-j4v5l\" (UID: \"49a22f36-fe11-4c4c-aa8d-19af1a508da2\") " pod="kserve/llmisvc-controller-manager-654c557856-j4v5l"
Jul 09 18:12:58.002486 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:58.002452    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49a22f36-fe11-4c4c-aa8d-19af1a508da2-cert\") pod \"llmisvc-controller-manager-654c557856-j4v5l\" (UID: \"49a22f36-fe11-4c4c-aa8d-19af1a508da2\") " pod="kserve/llmisvc-controller-manager-654c557856-j4v5l"
Jul 09 18:12:58.002694 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:58.002501    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8e5590f-b1e4-4b7f-a109-571830c28dfe-cert\") pod \"kserve-controller-manager-748bdf6488-fr588\" (UID: \"f8e5590f-b1e4-4b7f-a109-571830c28dfe\") " pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:12:58.004819 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:58.004790    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49a22f36-fe11-4c4c-aa8d-19af1a508da2-cert\") pod \"llmisvc-controller-manager-654c557856-j4v5l\" (UID: \"49a22f36-fe11-4c4c-aa8d-19af1a508da2\") " pod="kserve/llmisvc-controller-manager-654c557856-j4v5l"
Jul 09 18:12:58.004958 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:58.004942    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8e5590f-b1e4-4b7f-a109-571830c28dfe-cert\") pod \"kserve-controller-manager-748bdf6488-fr588\" (UID: \"f8e5590f-b1e4-4b7f-a109-571830c28dfe\") " pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:12:58.202137 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:58.202064    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:12:58.216956 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:58.216936    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-j4v5l"
Jul 09 18:12:58.334527 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:58.334493    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-fr588"]
Jul 09 18:12:58.337429 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:12:58.337403    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e5590f_b1e4_4b7f_a109_571830c28dfe.slice/crio-13abee28979764cf060db6d8dff018ca186d54955a733b1d762be11e5a20fe55 WatchSource:0}: Error finding container 13abee28979764cf060db6d8dff018ca186d54955a733b1d762be11e5a20fe55: Status 404 returned error can't find the container with id 13abee28979764cf060db6d8dff018ca186d54955a733b1d762be11e5a20fe55
Jul 09 18:12:58.351667 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:58.351644    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-j4v5l"]
Jul 09 18:12:58.353805 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:12:58.353780    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod49a22f36_fe11_4c4c_aa8d_19af1a508da2.slice/crio-96b8f739595235f318c568c73e1c417eb361ee4ba78584318decd40486d1180f WatchSource:0}: Error finding container 96b8f739595235f318c568c73e1c417eb361ee4ba78584318decd40486d1180f: Status 404 returned error can't find the container with id 96b8f739595235f318c568c73e1c417eb361ee4ba78584318decd40486d1180f
Jul 09 18:12:58.955045 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:58.955000    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-j4v5l" event={"ID":"49a22f36-fe11-4c4c-aa8d-19af1a508da2","Type":"ContainerStarted","Data":"96b8f739595235f318c568c73e1c417eb361ee4ba78584318decd40486d1180f"}
Jul 09 18:12:58.956402 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:12:58.956339    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-fr588" event={"ID":"f8e5590f-b1e4-4b7f-a109-571830c28dfe","Type":"ContainerStarted","Data":"13abee28979764cf060db6d8dff018ca186d54955a733b1d762be11e5a20fe55"}
Jul 09 18:13:02.973145 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:02.973111    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-j4v5l" event={"ID":"49a22f36-fe11-4c4c-aa8d-19af1a508da2","Type":"ContainerStarted","Data":"4c43be4f45cf3b20ac9b24589a7bdbe1d29f682d563911fc5197648e2828af0b"}
Jul 09 18:13:02.973530 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:02.973227    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-j4v5l"
Jul 09 18:13:02.974604 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:02.974581    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-fr588" event={"ID":"f8e5590f-b1e4-4b7f-a109-571830c28dfe","Type":"ContainerStarted","Data":"7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d"}
Jul 09 18:13:02.974709 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:02.974636    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:13:02.991661 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:02.991611    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-j4v5l" podStartSLOduration=2.254141959 podStartE2EDuration="5.991594363s" totalImagesPullingTime="3.737452404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:02.991430292 +0000 UTC m=+537.325294373" watchObservedRunningTime="2026-07-09 18:13:02.991594363 +0000 UTC m=+537.325458441"
Jul 09 18:13:03.007080 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:03.007033    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-748bdf6488-fr588" podStartSLOduration=3.027505069 podStartE2EDuration="6.007018848s" totalImagesPullingTime="2.979513779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:03.006321294 +0000 UTC m=+537.340185371" watchObservedRunningTime="2026-07-09 18:13:03.007018848 +0000 UTC m=+537.340882926"
Jul 09 18:13:33.980578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:33.980548    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-j4v5l"
Jul 09 18:13:33.983640 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:33.983620    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:13:35.250069 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.250038    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-fr588"]
Jul 09 18:13:35.250495 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.250206    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-748bdf6488-fr588" podUID="f8e5590f-b1e4-4b7f-a109-571830c28dfe" containerName="manager" containerID="cri-o://7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d" gracePeriod=10
Jul 09 18:13:35.275156 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.275133    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-jq2q7"]
Jul 09 18:13:35.276950 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.276934    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-jq2q7"
Jul 09 18:13:35.287633 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.287608    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-jq2q7"]
Jul 09 18:13:35.371500 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.371473    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9xx\" (UniqueName: \"kubernetes.io/projected/5b3ebd34-180d-40e4-943f-47105daf1629-kube-api-access-2c9xx\") pod \"kserve-controller-manager-748bdf6488-jq2q7\" (UID: \"5b3ebd34-180d-40e4-943f-47105daf1629\") " pod="kserve/kserve-controller-manager-748bdf6488-jq2q7"
Jul 09 18:13:35.371612 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.371533    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b3ebd34-180d-40e4-943f-47105daf1629-cert\") pod \"kserve-controller-manager-748bdf6488-jq2q7\" (UID: \"5b3ebd34-180d-40e4-943f-47105daf1629\") " pod="kserve/kserve-controller-manager-748bdf6488-jq2q7"
Jul 09 18:13:35.472377 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.472347    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b3ebd34-180d-40e4-943f-47105daf1629-cert\") pod \"kserve-controller-manager-748bdf6488-jq2q7\" (UID: \"5b3ebd34-180d-40e4-943f-47105daf1629\") " pod="kserve/kserve-controller-manager-748bdf6488-jq2q7"
Jul 09 18:13:35.472504 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.472408    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9xx\" (UniqueName: \"kubernetes.io/projected/5b3ebd34-180d-40e4-943f-47105daf1629-kube-api-access-2c9xx\") pod \"kserve-controller-manager-748bdf6488-jq2q7\" (UID: \"5b3ebd34-180d-40e4-943f-47105daf1629\") " pod="kserve/kserve-controller-manager-748bdf6488-jq2q7"
Jul 09 18:13:35.474936 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.474916    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b3ebd34-180d-40e4-943f-47105daf1629-cert\") pod \"kserve-controller-manager-748bdf6488-jq2q7\" (UID: \"5b3ebd34-180d-40e4-943f-47105daf1629\") " pod="kserve/kserve-controller-manager-748bdf6488-jq2q7"
Jul 09 18:13:35.481068 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.481048    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9xx\" (UniqueName: \"kubernetes.io/projected/5b3ebd34-180d-40e4-943f-47105daf1629-kube-api-access-2c9xx\") pod \"kserve-controller-manager-748bdf6488-jq2q7\" (UID: \"5b3ebd34-180d-40e4-943f-47105daf1629\") " pod="kserve/kserve-controller-manager-748bdf6488-jq2q7"
Jul 09 18:13:35.491821 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.491803    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:13:35.573327 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.573300    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxmp2\" (UniqueName: \"kubernetes.io/projected/f8e5590f-b1e4-4b7f-a109-571830c28dfe-kube-api-access-sxmp2\") pod \"f8e5590f-b1e4-4b7f-a109-571830c28dfe\" (UID: \"f8e5590f-b1e4-4b7f-a109-571830c28dfe\") "
Jul 09 18:13:35.573486 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.573340    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8e5590f-b1e4-4b7f-a109-571830c28dfe-cert\") pod \"f8e5590f-b1e4-4b7f-a109-571830c28dfe\" (UID: \"f8e5590f-b1e4-4b7f-a109-571830c28dfe\") "
Jul 09 18:13:35.575466 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.575443    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e5590f-b1e4-4b7f-a109-571830c28dfe-cert" (OuterVolumeSpecName: "cert") pod "f8e5590f-b1e4-4b7f-a109-571830c28dfe" (UID: "f8e5590f-b1e4-4b7f-a109-571830c28dfe"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:13:35.575535 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.575483    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e5590f-b1e4-4b7f-a109-571830c28dfe-kube-api-access-sxmp2" (OuterVolumeSpecName: "kube-api-access-sxmp2") pod "f8e5590f-b1e4-4b7f-a109-571830c28dfe" (UID: "f8e5590f-b1e4-4b7f-a109-571830c28dfe"). InnerVolumeSpecName "kube-api-access-sxmp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:13:35.618760 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.618731    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-jq2q7"
Jul 09 18:13:35.674623 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.674593    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxmp2\" (UniqueName: \"kubernetes.io/projected/f8e5590f-b1e4-4b7f-a109-571830c28dfe-kube-api-access-sxmp2\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:13:35.674623 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.674619    2568 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8e5590f-b1e4-4b7f-a109-571830c28dfe-cert\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:13:35.742312 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:35.742281    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-jq2q7"]
Jul 09 18:13:35.745428 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:13:35.745400    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3ebd34_180d_40e4_943f_47105daf1629.slice/crio-02db940551fd6e7c71cd717a3c267fc7160c1dab434c6d006e4ff276a762ca53 WatchSource:0}: Error finding container 02db940551fd6e7c71cd717a3c267fc7160c1dab434c6d006e4ff276a762ca53: Status 404 returned error can't find the container with id 02db940551fd6e7c71cd717a3c267fc7160c1dab434c6d006e4ff276a762ca53
Jul 09 18:13:36.085061 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:36.085026    2568 generic.go:356] "Generic (PLEG): container finished" podID="f8e5590f-b1e4-4b7f-a109-571830c28dfe" containerID="7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d" exitCode=0
Jul 09 18:13:36.085259 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:36.085070    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-748bdf6488-fr588"
Jul 09 18:13:36.085259 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:36.085097    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-fr588" event={"ID":"f8e5590f-b1e4-4b7f-a109-571830c28dfe","Type":"ContainerDied","Data":"7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d"}
Jul 09 18:13:36.085259 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:36.085146    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-fr588" event={"ID":"f8e5590f-b1e4-4b7f-a109-571830c28dfe","Type":"ContainerDied","Data":"13abee28979764cf060db6d8dff018ca186d54955a733b1d762be11e5a20fe55"}
Jul 09 18:13:36.085259 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:36.085174    2568 scope.go:117] "RemoveContainer" containerID="7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d"
Jul 09 18:13:36.086382 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:36.086356    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-jq2q7" event={"ID":"5b3ebd34-180d-40e4-943f-47105daf1629","Type":"ContainerStarted","Data":"02db940551fd6e7c71cd717a3c267fc7160c1dab434c6d006e4ff276a762ca53"}
Jul 09 18:13:36.092991 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:36.092956    2568 scope.go:117] "RemoveContainer" containerID="7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d"
Jul 09 18:13:36.093260 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:13:36.093239    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d\": container with ID starting with 7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d not found: ID does not exist" containerID="7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d"
Jul 09 18:13:36.093330 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:36.093267    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d"} err="failed to get container status \"7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d\": rpc error: code = NotFound desc = could not find container \"7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d\": container with ID starting with 7b80b3c345053a55b9a1a5444870cb7630c1934a37ce928564b884ceb5f62d9d not found: ID does not exist"
Jul 09 18:13:36.106530 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:36.106508    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-fr588"]
Jul 09 18:13:36.111570 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:36.111551    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-748bdf6488-fr588"]
Jul 09 18:13:36.225524 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:36.225492    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e5590f-b1e4-4b7f-a109-571830c28dfe" path="/var/lib/kubelet/pods/f8e5590f-b1e4-4b7f-a109-571830c28dfe/volumes"
Jul 09 18:13:37.091609 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:37.091574    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-748bdf6488-jq2q7" event={"ID":"5b3ebd34-180d-40e4-943f-47105daf1629","Type":"ContainerStarted","Data":"15575a9856e9762bb22061bed90bdf2d138c25878beb652f3210cb9a0aca709a"}
Jul 09 18:13:37.092020 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:37.091691    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-748bdf6488-jq2q7"
Jul 09 18:13:37.111203 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:13:37.111151    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-748bdf6488-jq2q7" podStartSLOduration=1.520085729 podStartE2EDuration="2.111129395s" totalImagesPullingTime="591.043666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:37.110995843 +0000 UTC m=+571.444859940" watchObservedRunningTime="2026-07-09 18:13:37.111129395 +0000 UTC m=+571.444993468"
Jul 09 18:14:06.128962 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:06.128935    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:14:06.130408 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:06.130390    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:14:08.099541 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:08.099512    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-748bdf6488-jq2q7"
Jul 09 18:14:09.195278 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.195246    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-9kvtv"]
Jul 09 18:14:09.195630 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.195496    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e5590f-b1e4-4b7f-a109-571830c28dfe" containerName="manager"
Jul 09 18:14:09.195630 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.195508    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e5590f-b1e4-4b7f-a109-571830c28dfe" containerName="manager"
Jul 09 18:14:09.195630 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.195550    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e5590f-b1e4-4b7f-a109-571830c28dfe" containerName="manager"
Jul 09 18:14:09.199493 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.199464    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-9kvtv"
Jul 09 18:14:09.202610 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.202595    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 09 18:14:09.202698 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.202661    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-8g2vj\""
Jul 09 18:14:09.209673 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.209653    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-9kvtv"]
Jul 09 18:14:09.213424 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.213403    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469358e6-fcef-42e1-9f26-5d97e1eba39f-tls-certs\") pod \"model-serving-api-fd65d7d6-9kvtv\" (UID: \"469358e6-fcef-42e1-9f26-5d97e1eba39f\") " pod="kserve/model-serving-api-fd65d7d6-9kvtv"
Jul 09 18:14:09.213524 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.213483    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvll4\" (UniqueName: \"kubernetes.io/projected/469358e6-fcef-42e1-9f26-5d97e1eba39f-kube-api-access-bvll4\") pod \"model-serving-api-fd65d7d6-9kvtv\" (UID: \"469358e6-fcef-42e1-9f26-5d97e1eba39f\") " pod="kserve/model-serving-api-fd65d7d6-9kvtv"
Jul 09 18:14:09.314655 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.314624    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvll4\" (UniqueName: \"kubernetes.io/projected/469358e6-fcef-42e1-9f26-5d97e1eba39f-kube-api-access-bvll4\") pod \"model-serving-api-fd65d7d6-9kvtv\" (UID: \"469358e6-fcef-42e1-9f26-5d97e1eba39f\") " pod="kserve/model-serving-api-fd65d7d6-9kvtv"
Jul 09 18:14:09.314833 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.314670    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469358e6-fcef-42e1-9f26-5d97e1eba39f-tls-certs\") pod \"model-serving-api-fd65d7d6-9kvtv\" (UID: \"469358e6-fcef-42e1-9f26-5d97e1eba39f\") " pod="kserve/model-serving-api-fd65d7d6-9kvtv"
Jul 09 18:14:09.317017 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.316994    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469358e6-fcef-42e1-9f26-5d97e1eba39f-tls-certs\") pod \"model-serving-api-fd65d7d6-9kvtv\" (UID: \"469358e6-fcef-42e1-9f26-5d97e1eba39f\") " pod="kserve/model-serving-api-fd65d7d6-9kvtv"
Jul 09 18:14:09.327307 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.327286    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvll4\" (UniqueName: \"kubernetes.io/projected/469358e6-fcef-42e1-9f26-5d97e1eba39f-kube-api-access-bvll4\") pod \"model-serving-api-fd65d7d6-9kvtv\" (UID: \"469358e6-fcef-42e1-9f26-5d97e1eba39f\") " pod="kserve/model-serving-api-fd65d7d6-9kvtv"
Jul 09 18:14:09.510139 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.510107    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-9kvtv"
Jul 09 18:14:09.631358 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:09.631327    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-9kvtv"]
Jul 09 18:14:09.634273 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:14:09.634238    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469358e6_fcef_42e1_9f26_5d97e1eba39f.slice/crio-1a620d7da6a893756197cabe54688f7cac2c64c4a4ebdfb4af1d96231d0277cc WatchSource:0}: Error finding container 1a620d7da6a893756197cabe54688f7cac2c64c4a4ebdfb4af1d96231d0277cc: Status 404 returned error can't find the container with id 1a620d7da6a893756197cabe54688f7cac2c64c4a4ebdfb4af1d96231d0277cc
Jul 09 18:14:10.209215 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:10.209179    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-9kvtv" event={"ID":"469358e6-fcef-42e1-9f26-5d97e1eba39f","Type":"ContainerStarted","Data":"1a620d7da6a893756197cabe54688f7cac2c64c4a4ebdfb4af1d96231d0277cc"}
Jul 09 18:14:12.225849 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:12.225810    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-9kvtv" event={"ID":"469358e6-fcef-42e1-9f26-5d97e1eba39f","Type":"ContainerStarted","Data":"8c384176c238f448c120ef5e03d74aaf7bb8889c480551bdfdd1b8c7eb4a0b31"}
Jul 09 18:14:12.226275 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:12.226029    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-9kvtv"
Jul 09 18:14:12.243949 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:12.243905    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-9kvtv" podStartSLOduration=1.564587511 podStartE2EDuration="3.243871794s" totalImagesPullingTime="1.679284283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:14:12.243018531 +0000 UTC m=+606.576882619" watchObservedRunningTime="2026-07-09 18:14:12.243871794 +0000 UTC m=+606.577735871"
Jul 09 18:14:23.233705 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:23.233675    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-9kvtv"
Jul 09 18:14:33.451516 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.451439    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f8455744f-n7gvk"]
Jul 09 18:14:33.454408 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.454386    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.457601 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.457577    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 18:14:33.457686 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.457577    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 18:14:33.458984 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.458965    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7bkfr\""
Jul 09 18:14:33.459073 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.458994    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 18:14:33.459073 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.458983    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 18:14:33.459073 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.458970    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 18:14:33.459352 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.459338    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 18:14:33.461199 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.461177    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 18:14:33.468018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.467130    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f8455744f-n7gvk"]
Jul 09 18:14:33.470944 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.470922    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 18:14:33.484237 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.484201    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c4910e1-16f9-41df-adca-6381c1685d4d-console-serving-cert\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.484337 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.484247    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-service-ca\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.484383 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.484327    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c4910e1-16f9-41df-adca-6381c1685d4d-console-oauth-config\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.484383 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.484365    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-trusted-ca-bundle\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.484464 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.484397    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n587x\" (UniqueName: \"kubernetes.io/projected/8c4910e1-16f9-41df-adca-6381c1685d4d-kube-api-access-n587x\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.484464 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.484421    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-console-config\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.484464 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.484445    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-oauth-serving-cert\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.585083 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.585038    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-service-ca\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.585257 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.585099    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c4910e1-16f9-41df-adca-6381c1685d4d-console-oauth-config\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.585257 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.585204    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-trusted-ca-bundle\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.585257 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.585257    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n587x\" (UniqueName: \"kubernetes.io/projected/8c4910e1-16f9-41df-adca-6381c1685d4d-kube-api-access-n587x\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.585413 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.585285    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-console-config\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.585413 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.585302    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-oauth-serving-cert\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.585413 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.585327    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c4910e1-16f9-41df-adca-6381c1685d4d-console-serving-cert\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.585857 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.585834    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-service-ca\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.586052 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.586034    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-console-config\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.586147 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.586122    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-oauth-serving-cert\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.586375 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.586352    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c4910e1-16f9-41df-adca-6381c1685d4d-trusted-ca-bundle\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.587652 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.587631    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c4910e1-16f9-41df-adca-6381c1685d4d-console-oauth-config\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.587773 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.587756    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c4910e1-16f9-41df-adca-6381c1685d4d-console-serving-cert\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.594254 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.594235    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n587x\" (UniqueName: \"kubernetes.io/projected/8c4910e1-16f9-41df-adca-6381c1685d4d-kube-api-access-n587x\") pod \"console-7f8455744f-n7gvk\" (UID: \"8c4910e1-16f9-41df-adca-6381c1685d4d\") " pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.764637 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.764606    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:33.891647 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:33.891619    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f8455744f-n7gvk"]
Jul 09 18:14:33.893497 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:14:33.893472    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4910e1_16f9_41df_adca_6381c1685d4d.slice/crio-7577a5f9b3f9780b6dba81566aa87021c8fc421557f616e1e97ee71e7cca4f4b WatchSource:0}: Error finding container 7577a5f9b3f9780b6dba81566aa87021c8fc421557f616e1e97ee71e7cca4f4b: Status 404 returned error can't find the container with id 7577a5f9b3f9780b6dba81566aa87021c8fc421557f616e1e97ee71e7cca4f4b
Jul 09 18:14:34.298540 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:34.298509    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8455744f-n7gvk" event={"ID":"8c4910e1-16f9-41df-adca-6381c1685d4d","Type":"ContainerStarted","Data":"e10ba186e4fac417f0c915b95d144d450da566a24a918c7f197f739f4b45e596"}
Jul 09 18:14:34.298719 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:34.298555    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8455744f-n7gvk" event={"ID":"8c4910e1-16f9-41df-adca-6381c1685d4d","Type":"ContainerStarted","Data":"7577a5f9b3f9780b6dba81566aa87021c8fc421557f616e1e97ee71e7cca4f4b"}
Jul 09 18:14:34.317128 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:34.317082    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f8455744f-n7gvk" podStartSLOduration=1.317066891 podStartE2EDuration="1.317066891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:14:34.316071841 +0000 UTC m=+628.649935920" watchObservedRunningTime="2026-07-09 18:14:34.317066891 +0000 UTC m=+628.650930969"
Jul 09 18:14:43.764762 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:43.764718    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:43.765447 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:43.764787    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:43.769530 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:43.769506    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:14:44.336742 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:14:44.336715    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f8455744f-n7gvk"
Jul 09 18:15:57.015009 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.014975    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"]
Jul 09 18:15:57.018528 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.018503    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.021196 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.021133    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6s7vj\""
Jul 09 18:15:57.021196 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.021130    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 09 18:15:57.021441 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.021426    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:15:57.022395 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.022375    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:15:57.022458 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.022445    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:15:57.022518 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.022449    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 09 18:15:57.022518 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.022504    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:15:57.027835 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.027812    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"]
Jul 09 18:15:57.100645 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.100604    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk75v\" (UniqueName: \"kubernetes.io/projected/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-api-access-wk75v\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.100834 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.100667    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.100834 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.100695    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9de496-a586-4911-8848-7c3b5c08ab4b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.100834 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.100715    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.100834 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.100762    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9de496-a586-4911-8848-7c3b5c08ab4b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.201476 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.201439    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.201476 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.201482    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9de496-a586-4911-8848-7c3b5c08ab4b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.201695 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.201504    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.201695 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.201534    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9de496-a586-4911-8848-7c3b5c08ab4b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.202343 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.202051    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk75v\" (UniqueName: \"kubernetes.io/projected/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-api-access-wk75v\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.202343 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:15:57.202090    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 09 18:15:57.202343 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:15:57.202180    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e9de496-a586-4911-8848-7c3b5c08ab4b-proxy-tls podName:3e9de496-a586-4911-8848-7c3b5c08ab4b nodeName:}" failed. No retries permitted until 2026-07-09 18:15:57.702156915 +0000 UTC m=+712.036020976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3e9de496-a586-4911-8848-7c3b5c08ab4b-proxy-tls") pod "isvc-sklearn-graph-2-predictor-656c946759-dflhh" (UID: "3e9de496-a586-4911-8848-7c3b5c08ab4b") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 09 18:15:57.202343 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.202214    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9de496-a586-4911-8848-7c3b5c08ab4b-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.206463 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.202728    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.206463 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.202755    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.211030 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.211003    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk75v\" (UniqueName: \"kubernetes.io/projected/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-api-access-wk75v\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.706986 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.706953    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9de496-a586-4911-8848-7c3b5c08ab4b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.709327 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.709301    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9de496-a586-4911-8848-7c3b5c08ab4b-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-656c946759-dflhh\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:57.929689 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:57.929646    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:15:58.060573 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:58.060551    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"]
Jul 09 18:15:58.062960 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:15:58.062931    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9de496_a586_4911_8848_7c3b5c08ab4b.slice/crio-e193918693aa7be72551803d4ee5fe346f6b946cd59aad5ea4f0eed2beb28bc8 WatchSource:0}: Error finding container e193918693aa7be72551803d4ee5fe346f6b946cd59aad5ea4f0eed2beb28bc8: Status 404 returned error can't find the container with id e193918693aa7be72551803d4ee5fe346f6b946cd59aad5ea4f0eed2beb28bc8
Jul 09 18:15:58.064792 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:58.064776    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:15:58.586211 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:15:58.586172    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" event={"ID":"3e9de496-a586-4911-8848-7c3b5c08ab4b","Type":"ContainerStarted","Data":"e193918693aa7be72551803d4ee5fe346f6b946cd59aad5ea4f0eed2beb28bc8"}
Jul 09 18:16:02.603162 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:02.603122    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" event={"ID":"3e9de496-a586-4911-8848-7c3b5c08ab4b","Type":"ContainerStarted","Data":"300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d"}
Jul 09 18:16:05.614783 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:05.614749    2568 generic.go:356] "Generic (PLEG): container finished" podID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerID="300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d" exitCode=0
Jul 09 18:16:05.614783 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:05.614785    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" event={"ID":"3e9de496-a586-4911-8848-7c3b5c08ab4b","Type":"ContainerDied","Data":"300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d"}
Jul 09 18:16:19.687560 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:19.687522    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" event={"ID":"3e9de496-a586-4911-8848-7c3b5c08ab4b","Type":"ContainerStarted","Data":"620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507"}
Jul 09 18:16:21.696907 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:21.696853    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" event={"ID":"3e9de496-a586-4911-8848-7c3b5c08ab4b","Type":"ContainerStarted","Data":"497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819"}
Jul 09 18:16:21.697301 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:21.697034    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:16:21.718032 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:21.717980    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podStartSLOduration=3.12826407 podStartE2EDuration="25.717966111s" totalImagesPullingTime="19.095183991s" totalInitContainerRuntime="3.49451805s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:21.716643632 +0000 UTC m=+736.050507710" watchObservedRunningTime="2026-07-09 18:16:21.717966111 +0000 UTC m=+736.051830228"
Jul 09 18:16:22.700448 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:22.700412    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:16:22.701538 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:22.701510    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:16:23.704235 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:23.704196    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:16:28.709405 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:28.709376    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:16:28.710031 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:28.710001    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:16:38.709898 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:38.709843    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:16:48.709989 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:48.709951    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:16:58.710031 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:16:58.709993    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:17:08.710040 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:08.710000    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:17:18.710790 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:18.710749    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:17:26.509117 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:26.509078    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"]
Jul 09 18:17:26.511323 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:26.511302    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:26.514346 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:26.514323    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-db4c0-serving-cert\""
Jul 09 18:17:26.514346 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:26.514340    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-db4c0-kube-rbac-proxy-sar-config\""
Jul 09 18:17:26.521283 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:26.521260    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"]
Jul 09 18:17:26.588540 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:26.588511    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcc6f802-bbd7-4be2-9f01-277246a7e73b-openshift-service-ca-bundle\") pod \"switch-graph-db4c0-65d6dcf86-bx2xk\" (UID: \"dcc6f802-bbd7-4be2-9f01-277246a7e73b\") " pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:26.588686 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:26.588578    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcc6f802-bbd7-4be2-9f01-277246a7e73b-proxy-tls\") pod \"switch-graph-db4c0-65d6dcf86-bx2xk\" (UID: \"dcc6f802-bbd7-4be2-9f01-277246a7e73b\") " pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:26.689095 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:26.689053    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcc6f802-bbd7-4be2-9f01-277246a7e73b-proxy-tls\") pod \"switch-graph-db4c0-65d6dcf86-bx2xk\" (UID: \"dcc6f802-bbd7-4be2-9f01-277246a7e73b\") " pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:26.689266 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:26.689118    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcc6f802-bbd7-4be2-9f01-277246a7e73b-openshift-service-ca-bundle\") pod \"switch-graph-db4c0-65d6dcf86-bx2xk\" (UID: \"dcc6f802-bbd7-4be2-9f01-277246a7e73b\") " pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:26.689266 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:17:26.689197    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-db4c0-serving-cert: secret "switch-graph-db4c0-serving-cert" not found
Jul 09 18:17:26.689386 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:17:26.689272    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc6f802-bbd7-4be2-9f01-277246a7e73b-proxy-tls podName:dcc6f802-bbd7-4be2-9f01-277246a7e73b nodeName:}" failed. No retries permitted until 2026-07-09 18:17:27.189255765 +0000 UTC m=+801.523119821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dcc6f802-bbd7-4be2-9f01-277246a7e73b-proxy-tls") pod "switch-graph-db4c0-65d6dcf86-bx2xk" (UID: "dcc6f802-bbd7-4be2-9f01-277246a7e73b") : secret "switch-graph-db4c0-serving-cert" not found
Jul 09 18:17:26.689787 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:26.689763    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcc6f802-bbd7-4be2-9f01-277246a7e73b-openshift-service-ca-bundle\") pod \"switch-graph-db4c0-65d6dcf86-bx2xk\" (UID: \"dcc6f802-bbd7-4be2-9f01-277246a7e73b\") " pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:27.192671 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:27.192619    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcc6f802-bbd7-4be2-9f01-277246a7e73b-proxy-tls\") pod \"switch-graph-db4c0-65d6dcf86-bx2xk\" (UID: \"dcc6f802-bbd7-4be2-9f01-277246a7e73b\") " pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:27.195182 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:27.195158    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcc6f802-bbd7-4be2-9f01-277246a7e73b-proxy-tls\") pod \"switch-graph-db4c0-65d6dcf86-bx2xk\" (UID: \"dcc6f802-bbd7-4be2-9f01-277246a7e73b\") " pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:27.421991 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:27.421953    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:27.545855 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:27.545824    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"]
Jul 09 18:17:27.548580 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:17:27.548553    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc6f802_bbd7_4be2_9f01_277246a7e73b.slice/crio-89ada244e53c093d9dbfa7ce4dfa63dd523cc1f1baebef1ce24891c4a612744f WatchSource:0}: Error finding container 89ada244e53c093d9dbfa7ce4dfa63dd523cc1f1baebef1ce24891c4a612744f: Status 404 returned error can't find the container with id 89ada244e53c093d9dbfa7ce4dfa63dd523cc1f1baebef1ce24891c4a612744f
Jul 09 18:17:27.930743 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:27.930708    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" event={"ID":"dcc6f802-bbd7-4be2-9f01-277246a7e73b","Type":"ContainerStarted","Data":"89ada244e53c093d9dbfa7ce4dfa63dd523cc1f1baebef1ce24891c4a612744f"}
Jul 09 18:17:28.710598 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:28.710552    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 09 18:17:30.943283 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:30.943205    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" event={"ID":"dcc6f802-bbd7-4be2-9f01-277246a7e73b","Type":"ContainerStarted","Data":"3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba"}
Jul 09 18:17:30.943628 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:30.943355    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:30.962181 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:30.962136    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" podStartSLOduration=1.949655252 podStartE2EDuration="4.96212157s" totalImagesPullingTime="3.012466318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:30.961303674 +0000 UTC m=+805.295167752" watchObservedRunningTime="2026-07-09 18:17:30.96212157 +0000 UTC m=+805.295985649"
Jul 09 18:17:34.226152 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:34.226126    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:17:36.952023 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:36.951996    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:40.659281 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:40.659247    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"]
Jul 09 18:17:40.659778 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:40.659442    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" podUID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" containerName="switch-graph-db4c0" containerID="cri-o://3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba" gracePeriod=30
Jul 09 18:17:41.951421 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:41.951385    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" podUID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" containerName="switch-graph-db4c0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:46.950446 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:46.950410    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" podUID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" containerName="switch-graph-db4c0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:51.950949 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:51.950909    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" podUID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" containerName="switch-graph-db4c0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:51.951293 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:51.951025    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:17:56.950500 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:17:56.950464    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" podUID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" containerName="switch-graph-db4c0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:01.951128 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:01.951084    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" podUID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" containerName="switch-graph-db4c0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:06.476042 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:06.476005    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"]
Jul 09 18:18:06.479507 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:06.479488    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:06.482208 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:06.482182    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 09 18:18:06.482208 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:06.482191    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 09 18:18:06.490311 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:06.490282    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"]
Jul 09 18:18:06.607269 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:06.607213    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/740d8dd7-a30d-4be8-a06a-b577c5b45320-proxy-tls\") pod \"model-chainer-c8944fcf6-s6ljt\" (UID: \"740d8dd7-a30d-4be8-a06a-b577c5b45320\") " pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:06.607439 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:06.607319    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/740d8dd7-a30d-4be8-a06a-b577c5b45320-openshift-service-ca-bundle\") pod \"model-chainer-c8944fcf6-s6ljt\" (UID: \"740d8dd7-a30d-4be8-a06a-b577c5b45320\") " pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:06.708695 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:06.708653    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/740d8dd7-a30d-4be8-a06a-b577c5b45320-proxy-tls\") pod \"model-chainer-c8944fcf6-s6ljt\" (UID: \"740d8dd7-a30d-4be8-a06a-b577c5b45320\") " pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:06.708844 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:06.708707    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/740d8dd7-a30d-4be8-a06a-b577c5b45320-openshift-service-ca-bundle\") pod \"model-chainer-c8944fcf6-s6ljt\" (UID: \"740d8dd7-a30d-4be8-a06a-b577c5b45320\") " pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:06.708844 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:18:06.708829    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 09 18:18:06.708986 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:18:06.708933    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/740d8dd7-a30d-4be8-a06a-b577c5b45320-proxy-tls podName:740d8dd7-a30d-4be8-a06a-b577c5b45320 nodeName:}" failed. No retries permitted until 2026-07-09 18:18:07.208909689 +0000 UTC m=+841.542773763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/740d8dd7-a30d-4be8-a06a-b577c5b45320-proxy-tls") pod "model-chainer-c8944fcf6-s6ljt" (UID: "740d8dd7-a30d-4be8-a06a-b577c5b45320") : secret "model-chainer-serving-cert" not found
Jul 09 18:18:06.709360 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:06.709339    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/740d8dd7-a30d-4be8-a06a-b577c5b45320-openshift-service-ca-bundle\") pod \"model-chainer-c8944fcf6-s6ljt\" (UID: \"740d8dd7-a30d-4be8-a06a-b577c5b45320\") " pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:06.950610 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:06.950573    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" podUID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" containerName="switch-graph-db4c0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:07.214458 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:07.214372    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/740d8dd7-a30d-4be8-a06a-b577c5b45320-proxy-tls\") pod \"model-chainer-c8944fcf6-s6ljt\" (UID: \"740d8dd7-a30d-4be8-a06a-b577c5b45320\") " pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:07.216821 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:07.216786    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/740d8dd7-a30d-4be8-a06a-b577c5b45320-proxy-tls\") pod \"model-chainer-c8944fcf6-s6ljt\" (UID: \"740d8dd7-a30d-4be8-a06a-b577c5b45320\") " pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:07.389838 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:07.389812    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:07.512370 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:07.512341    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"]
Jul 09 18:18:07.515946 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:18:07.515920    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740d8dd7_a30d_4be8_a06a_b577c5b45320.slice/crio-25406eb42f8fe470c3e6cbd1b4b2944a7489403598146059efed4c4bf74817e9 WatchSource:0}: Error finding container 25406eb42f8fe470c3e6cbd1b4b2944a7489403598146059efed4c4bf74817e9: Status 404 returned error can't find the container with id 25406eb42f8fe470c3e6cbd1b4b2944a7489403598146059efed4c4bf74817e9
Jul 09 18:18:08.072235 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:08.072197    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" event={"ID":"740d8dd7-a30d-4be8-a06a-b577c5b45320","Type":"ContainerStarted","Data":"3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402"}
Jul 09 18:18:08.072442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:08.072251    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" event={"ID":"740d8dd7-a30d-4be8-a06a-b577c5b45320","Type":"ContainerStarted","Data":"25406eb42f8fe470c3e6cbd1b4b2944a7489403598146059efed4c4bf74817e9"}
Jul 09 18:18:08.072442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:08.072314    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:08.089841 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:08.089792    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" podStartSLOduration=2.089778389 podStartE2EDuration="2.089778389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:08.08836157 +0000 UTC m=+842.422225649" watchObservedRunningTime="2026-07-09 18:18:08.089778389 +0000 UTC m=+842.423642466"
Jul 09 18:18:10.835287 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:10.835264    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:18:10.844195 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:10.844170    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcc6f802-bbd7-4be2-9f01-277246a7e73b-openshift-service-ca-bundle\") pod \"dcc6f802-bbd7-4be2-9f01-277246a7e73b\" (UID: \"dcc6f802-bbd7-4be2-9f01-277246a7e73b\") "
Jul 09 18:18:10.844508 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:10.844489    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc6f802-bbd7-4be2-9f01-277246a7e73b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "dcc6f802-bbd7-4be2-9f01-277246a7e73b" (UID: "dcc6f802-bbd7-4be2-9f01-277246a7e73b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:10.945269 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:10.945178    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcc6f802-bbd7-4be2-9f01-277246a7e73b-proxy-tls\") pod \"dcc6f802-bbd7-4be2-9f01-277246a7e73b\" (UID: \"dcc6f802-bbd7-4be2-9f01-277246a7e73b\") "
Jul 09 18:18:10.945428 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:10.945303    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcc6f802-bbd7-4be2-9f01-277246a7e73b-openshift-service-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:18:10.947341 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:10.947320    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc6f802-bbd7-4be2-9f01-277246a7e73b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dcc6f802-bbd7-4be2-9f01-277246a7e73b" (UID: "dcc6f802-bbd7-4be2-9f01-277246a7e73b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:11.045758 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:11.045729    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcc6f802-bbd7-4be2-9f01-277246a7e73b-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:18:11.090537 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:11.090493    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" event={"ID":"dcc6f802-bbd7-4be2-9f01-277246a7e73b","Type":"ContainerDied","Data":"3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba"}
Jul 09 18:18:11.090708 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:11.090558    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"
Jul 09 18:18:11.090708 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:11.090497    2568 generic.go:356] "Generic (PLEG): container finished" podID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" containerID="3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba" exitCode=137
Jul 09 18:18:11.090708 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:11.090596    2568 scope.go:117] "RemoveContainer" containerID="3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba"
Jul 09 18:18:11.090708 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:11.090696    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk" event={"ID":"dcc6f802-bbd7-4be2-9f01-277246a7e73b","Type":"ContainerDied","Data":"89ada244e53c093d9dbfa7ce4dfa63dd523cc1f1baebef1ce24891c4a612744f"}
Jul 09 18:18:11.099281 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:11.099263    2568 scope.go:117] "RemoveContainer" containerID="3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba"
Jul 09 18:18:11.099523 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:18:11.099505    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba\": container with ID starting with 3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba not found: ID does not exist" containerID="3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba"
Jul 09 18:18:11.099569 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:11.099531    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba"} err="failed to get container status \"3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba\": rpc error: code = NotFound desc = could not find container \"3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba\": container with ID starting with 3ca09bbf5e0c8630da98b9024ee0d73a29b481970bfda9a65add3b1613728eba not found: ID does not exist"
Jul 09 18:18:11.112240 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:11.112215    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"]
Jul 09 18:18:11.116335 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:11.116315    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-db4c0-65d6dcf86-bx2xk"]
Jul 09 18:18:12.226284 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:12.226252    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" path="/var/lib/kubelet/pods/dcc6f802-bbd7-4be2-9f01-277246a7e73b/volumes"
Jul 09 18:18:14.082211 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:14.082180    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:16.581212 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:16.581179    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"]
Jul 09 18:18:16.581565 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:16.581356    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" podUID="740d8dd7-a30d-4be8-a06a-b577c5b45320" containerName="model-chainer" containerID="cri-o://3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402" gracePeriod=30
Jul 09 18:18:16.740111 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:16.740077    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"]
Jul 09 18:18:16.740562 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:16.740444    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kube-rbac-proxy" containerID="cri-o://497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819" gracePeriod=30
Jul 09 18:18:16.740713 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:16.740572    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container" containerID="cri-o://620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507" gracePeriod=30
Jul 09 18:18:17.120291 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:17.120199    2568 generic.go:356] "Generic (PLEG): container finished" podID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerID="497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819" exitCode=2
Jul 09 18:18:17.120291 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:17.120272    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" event={"ID":"3e9de496-a586-4911-8848-7c3b5c08ab4b","Type":"ContainerDied","Data":"497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819"}
Jul 09 18:18:18.705314 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:18.705274    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 09 18:18:19.080409 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:19.080368    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" podUID="740d8dd7-a30d-4be8-a06a-b577c5b45320" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:21.289840 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.289814    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:18:21.321946 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.321876    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9de496-a586-4911-8848-7c3b5c08ab4b-proxy-tls\") pod \"3e9de496-a586-4911-8848-7c3b5c08ab4b\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") "
Jul 09 18:18:21.322125 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.321962    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk75v\" (UniqueName: \"kubernetes.io/projected/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-api-access-wk75v\") pod \"3e9de496-a586-4911-8848-7c3b5c08ab4b\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") "
Jul 09 18:18:21.322125 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.322017    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-cabundle-cert\") pod \"3e9de496-a586-4911-8848-7c3b5c08ab4b\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") "
Jul 09 18:18:21.322125 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.322088    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9de496-a586-4911-8848-7c3b5c08ab4b-kserve-provision-location\") pod \"3e9de496-a586-4911-8848-7c3b5c08ab4b\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") "
Jul 09 18:18:21.322125 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.322125    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-rbac-proxy-sar-config\") pod \"3e9de496-a586-4911-8848-7c3b5c08ab4b\" (UID: \"3e9de496-a586-4911-8848-7c3b5c08ab4b\") "
Jul 09 18:18:21.322454 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.322420    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9de496-a586-4911-8848-7c3b5c08ab4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e9de496-a586-4911-8848-7c3b5c08ab4b" (UID: "3e9de496-a586-4911-8848-7c3b5c08ab4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:21.322591 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.322533    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e9de496-a586-4911-8848-7c3b5c08ab4b" (UID: "3e9de496-a586-4911-8848-7c3b5c08ab4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:21.322658 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.322615    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e9de496-a586-4911-8848-7c3b5c08ab4b" (UID: "3e9de496-a586-4911-8848-7c3b5c08ab4b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:21.324091 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.324065    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-api-access-wk75v" (OuterVolumeSpecName: "kube-api-access-wk75v") pod "3e9de496-a586-4911-8848-7c3b5c08ab4b" (UID: "3e9de496-a586-4911-8848-7c3b5c08ab4b"). InnerVolumeSpecName "kube-api-access-wk75v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:21.324179 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.324099    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9de496-a586-4911-8848-7c3b5c08ab4b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e9de496-a586-4911-8848-7c3b5c08ab4b" (UID: "3e9de496-a586-4911-8848-7c3b5c08ab4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:21.423505 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.423427    2568 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-cabundle-cert\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:18:21.423505 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.423453    2568 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9de496-a586-4911-8848-7c3b5c08ab4b-kserve-provision-location\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:18:21.423505 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.423462    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:18:21.423505 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.423472    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9de496-a586-4911-8848-7c3b5c08ab4b-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:18:21.423505 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:21.423481    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk75v\" (UniqueName: \"kubernetes.io/projected/3e9de496-a586-4911-8848-7c3b5c08ab4b-kube-api-access-wk75v\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:18:22.141608 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.141565    2568 generic.go:356] "Generic (PLEG): container finished" podID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerID="620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507" exitCode=0
Jul 09 18:18:22.141775 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.141616    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" event={"ID":"3e9de496-a586-4911-8848-7c3b5c08ab4b","Type":"ContainerDied","Data":"620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507"}
Jul 09 18:18:22.141775 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.141628    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"
Jul 09 18:18:22.141775 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.141668    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh" event={"ID":"3e9de496-a586-4911-8848-7c3b5c08ab4b","Type":"ContainerDied","Data":"e193918693aa7be72551803d4ee5fe346f6b946cd59aad5ea4f0eed2beb28bc8"}
Jul 09 18:18:22.141775 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.141699    2568 scope.go:117] "RemoveContainer" containerID="497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819"
Jul 09 18:18:22.150003 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.149980    2568 scope.go:117] "RemoveContainer" containerID="620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507"
Jul 09 18:18:22.156738 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.156721    2568 scope.go:117] "RemoveContainer" containerID="300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d"
Jul 09 18:18:22.163379 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.163359    2568 scope.go:117] "RemoveContainer" containerID="497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819"
Jul 09 18:18:22.163599 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:18:22.163583    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819\": container with ID starting with 497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819 not found: ID does not exist" containerID="497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819"
Jul 09 18:18:22.163647 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.163607    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819"} err="failed to get container status \"497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819\": rpc error: code = NotFound desc = could not find container \"497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819\": container with ID starting with 497610f5535e861f38fee680203e4d76502c2030f5671c82d5bb70962b549819 not found: ID does not exist"
Jul 09 18:18:22.163647 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.163623    2568 scope.go:117] "RemoveContainer" containerID="620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507"
Jul 09 18:18:22.163881 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:18:22.163859    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507\": container with ID starting with 620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507 not found: ID does not exist" containerID="620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507"
Jul 09 18:18:22.163986 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.163963    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507"} err="failed to get container status \"620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507\": rpc error: code = NotFound desc = could not find container \"620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507\": container with ID starting with 620478afef6f5c187ab83c54c3fa9e6cd3a4bd46519ac726f146e26aa06c2507 not found: ID does not exist"
Jul 09 18:18:22.164025 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.163990    2568 scope.go:117] "RemoveContainer" containerID="300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d"
Jul 09 18:18:22.164154 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.164139    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"]
Jul 09 18:18:22.164233 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:18:22.164217    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d\": container with ID starting with 300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d not found: ID does not exist" containerID="300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d"
Jul 09 18:18:22.164275 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.164238    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d"} err="failed to get container status \"300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d\": rpc error: code = NotFound desc = could not find container \"300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d\": container with ID starting with 300113f13d1c0289960816b3eab99092826aa05f081931e30f9f3cb70248721d not found: ID does not exist"
Jul 09 18:18:22.168307 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.168287    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-656c946759-dflhh"]
Jul 09 18:18:22.226425 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:22.226397    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" path="/var/lib/kubelet/pods/3e9de496-a586-4911-8848-7c3b5c08ab4b/volumes"
Jul 09 18:18:24.080427 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:24.080388    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" podUID="740d8dd7-a30d-4be8-a06a-b577c5b45320" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:29.080897 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:29.080849    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" podUID="740d8dd7-a30d-4be8-a06a-b577c5b45320" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:29.081281 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:29.080974    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:34.080130 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:34.080089    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" podUID="740d8dd7-a30d-4be8-a06a-b577c5b45320" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:39.080338 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:39.080290    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" podUID="740d8dd7-a30d-4be8-a06a-b577c5b45320" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:40.919801 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.919767    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"]
Jul 09 18:18:40.920169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.920061    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="storage-initializer"
Jul 09 18:18:40.920169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.920073    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="storage-initializer"
Jul 09 18:18:40.920169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.920084    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container"
Jul 09 18:18:40.920169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.920090    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container"
Jul 09 18:18:40.920169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.920104    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kube-rbac-proxy"
Jul 09 18:18:40.920169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.920110    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kube-rbac-proxy"
Jul 09 18:18:40.920169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.920116    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" containerName="switch-graph-db4c0"
Jul 09 18:18:40.920169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.920121    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" containerName="switch-graph-db4c0"
Jul 09 18:18:40.920169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.920157    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kube-rbac-proxy"
Jul 09 18:18:40.920169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.920164    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcc6f802-bbd7-4be2-9f01-277246a7e73b" containerName="switch-graph-db4c0"
Jul 09 18:18:40.920169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.920172    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9de496-a586-4911-8848-7c3b5c08ab4b" containerName="kserve-container"
Jul 09 18:18:40.922944 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.922924    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:18:40.925762 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.925742    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-2a7bf-serving-cert\""
Jul 09 18:18:40.925861 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.925766    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-2a7bf-kube-rbac-proxy-sar-config\""
Jul 09 18:18:40.936404 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.936383    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"]
Jul 09 18:18:40.968954 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.968923    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04243aba-085d-469a-938c-db9675cc3ab5-openshift-service-ca-bundle\") pod \"switch-graph-2a7bf-59d54ccb5d-7s4cj\" (UID: \"04243aba-085d-469a-938c-db9675cc3ab5\") " pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:18:40.969072 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:40.968971    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04243aba-085d-469a-938c-db9675cc3ab5-proxy-tls\") pod \"switch-graph-2a7bf-59d54ccb5d-7s4cj\" (UID: \"04243aba-085d-469a-938c-db9675cc3ab5\") " pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:18:41.069961 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:41.069925    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04243aba-085d-469a-938c-db9675cc3ab5-openshift-service-ca-bundle\") pod \"switch-graph-2a7bf-59d54ccb5d-7s4cj\" (UID: \"04243aba-085d-469a-938c-db9675cc3ab5\") " pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:18:41.070112 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:41.069983    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04243aba-085d-469a-938c-db9675cc3ab5-proxy-tls\") pod \"switch-graph-2a7bf-59d54ccb5d-7s4cj\" (UID: \"04243aba-085d-469a-938c-db9675cc3ab5\") " pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:18:41.070112 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:18:41.070088    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-2a7bf-serving-cert: secret "switch-graph-2a7bf-serving-cert" not found
Jul 09 18:18:41.070182 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:18:41.070151    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04243aba-085d-469a-938c-db9675cc3ab5-proxy-tls podName:04243aba-085d-469a-938c-db9675cc3ab5 nodeName:}" failed. No retries permitted until 2026-07-09 18:18:41.570134688 +0000 UTC m=+875.903998744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/04243aba-085d-469a-938c-db9675cc3ab5-proxy-tls") pod "switch-graph-2a7bf-59d54ccb5d-7s4cj" (UID: "04243aba-085d-469a-938c-db9675cc3ab5") : secret "switch-graph-2a7bf-serving-cert" not found
Jul 09 18:18:41.070583 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:41.070550    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04243aba-085d-469a-938c-db9675cc3ab5-openshift-service-ca-bundle\") pod \"switch-graph-2a7bf-59d54ccb5d-7s4cj\" (UID: \"04243aba-085d-469a-938c-db9675cc3ab5\") " pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:18:41.573814 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:41.573776    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04243aba-085d-469a-938c-db9675cc3ab5-proxy-tls\") pod \"switch-graph-2a7bf-59d54ccb5d-7s4cj\" (UID: \"04243aba-085d-469a-938c-db9675cc3ab5\") " pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:18:41.576200 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:41.576173    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04243aba-085d-469a-938c-db9675cc3ab5-proxy-tls\") pod \"switch-graph-2a7bf-59d54ccb5d-7s4cj\" (UID: \"04243aba-085d-469a-938c-db9675cc3ab5\") " pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:18:41.832869 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:41.832770    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:18:41.961115 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:41.961091    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"]
Jul 09 18:18:41.963450 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:18:41.963425    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04243aba_085d_469a_938c_db9675cc3ab5.slice/crio-848cdd6b08bf11c77daca0e8e7670a7d755213a17528422b0e4f2c0e1fcb9f51 WatchSource:0}: Error finding container 848cdd6b08bf11c77daca0e8e7670a7d755213a17528422b0e4f2c0e1fcb9f51: Status 404 returned error can't find the container with id 848cdd6b08bf11c77daca0e8e7670a7d755213a17528422b0e4f2c0e1fcb9f51
Jul 09 18:18:42.214707 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:42.214621    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" event={"ID":"04243aba-085d-469a-938c-db9675cc3ab5","Type":"ContainerStarted","Data":"58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645"}
Jul 09 18:18:42.214707 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:42.214671    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" event={"ID":"04243aba-085d-469a-938c-db9675cc3ab5","Type":"ContainerStarted","Data":"848cdd6b08bf11c77daca0e8e7670a7d755213a17528422b0e4f2c0e1fcb9f51"}
Jul 09 18:18:42.214874 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:42.214748    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:18:42.235508 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:42.235460    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" podStartSLOduration=2.235446793 podStartE2EDuration="2.235446793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:42.233540042 +0000 UTC m=+876.567404120" watchObservedRunningTime="2026-07-09 18:18:42.235446793 +0000 UTC m=+876.569310871"
Jul 09 18:18:44.080397 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:44.080358    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" podUID="740d8dd7-a30d-4be8-a06a-b577c5b45320" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:46.743429 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:46.743401    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:46.824696 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:46.824662    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/740d8dd7-a30d-4be8-a06a-b577c5b45320-proxy-tls\") pod \"740d8dd7-a30d-4be8-a06a-b577c5b45320\" (UID: \"740d8dd7-a30d-4be8-a06a-b577c5b45320\") "
Jul 09 18:18:46.824966 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:46.824768    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/740d8dd7-a30d-4be8-a06a-b577c5b45320-openshift-service-ca-bundle\") pod \"740d8dd7-a30d-4be8-a06a-b577c5b45320\" (UID: \"740d8dd7-a30d-4be8-a06a-b577c5b45320\") "
Jul 09 18:18:46.825135 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:46.825109    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740d8dd7-a30d-4be8-a06a-b577c5b45320-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "740d8dd7-a30d-4be8-a06a-b577c5b45320" (UID: "740d8dd7-a30d-4be8-a06a-b577c5b45320"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:46.826703 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:46.826682    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/740d8dd7-a30d-4be8-a06a-b577c5b45320-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "740d8dd7-a30d-4be8-a06a-b577c5b45320" (UID: "740d8dd7-a30d-4be8-a06a-b577c5b45320"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:46.925993 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:46.925872    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/740d8dd7-a30d-4be8-a06a-b577c5b45320-openshift-service-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:18:46.925993 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:46.925938    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/740d8dd7-a30d-4be8-a06a-b577c5b45320-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:18:47.232439 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:47.232343    2568 generic.go:356] "Generic (PLEG): container finished" podID="740d8dd7-a30d-4be8-a06a-b577c5b45320" containerID="3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402" exitCode=0
Jul 09 18:18:47.232439 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:47.232388    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"
Jul 09 18:18:47.232661 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:47.232393    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" event={"ID":"740d8dd7-a30d-4be8-a06a-b577c5b45320","Type":"ContainerDied","Data":"3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402"}
Jul 09 18:18:47.232661 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:47.232523    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt" event={"ID":"740d8dd7-a30d-4be8-a06a-b577c5b45320","Type":"ContainerDied","Data":"25406eb42f8fe470c3e6cbd1b4b2944a7489403598146059efed4c4bf74817e9"}
Jul 09 18:18:47.232661 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:47.232548    2568 scope.go:117] "RemoveContainer" containerID="3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402"
Jul 09 18:18:47.240938 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:47.240917    2568 scope.go:117] "RemoveContainer" containerID="3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402"
Jul 09 18:18:47.241183 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:18:47.241163    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402\": container with ID starting with 3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402 not found: ID does not exist" containerID="3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402"
Jul 09 18:18:47.241254 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:47.241196    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402"} err="failed to get container status \"3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402\": rpc error: code = NotFound desc = could not find container \"3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402\": container with ID starting with 3a12d11b86d5b76de5f82107323f929619cf82a69179fdf85a0d30cb4dabb402 not found: ID does not exist"
Jul 09 18:18:47.253744 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:47.253720    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"]
Jul 09 18:18:47.257161 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:47.257140    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-c8944fcf6-s6ljt"]
Jul 09 18:18:48.226554 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:48.226524    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="740d8dd7-a30d-4be8-a06a-b577c5b45320" path="/var/lib/kubelet/pods/740d8dd7-a30d-4be8-a06a-b577c5b45320/volumes"
Jul 09 18:18:48.226925 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:18:48.226782    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:19:06.150047 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:06.149970    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:19:06.152415 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:06.152392    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:19:16.824747 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.824710    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"]
Jul 09 18:19:16.825194 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.825014    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740d8dd7-a30d-4be8-a06a-b577c5b45320" containerName="model-chainer"
Jul 09 18:19:16.825194 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.825026    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="740d8dd7-a30d-4be8-a06a-b577c5b45320" containerName="model-chainer"
Jul 09 18:19:16.825194 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.825083    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="740d8dd7-a30d-4be8-a06a-b577c5b45320" containerName="model-chainer"
Jul 09 18:19:16.829520 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.829491    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:19:16.832186 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.832166    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-e8119-serving-cert\""
Jul 09 18:19:16.832582 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.832555    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-e8119-kube-rbac-proxy-sar-config\""
Jul 09 18:19:16.844545 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.844526    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"]
Jul 09 18:19:16.848959 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.848938    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74551749-3516-461d-81d5-8c4e2cd54411-proxy-tls\") pod \"sequence-graph-e8119-7d78b5bfcc-x5zj4\" (UID: \"74551749-3516-461d-81d5-8c4e2cd54411\") " pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:19:16.849040 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.848971    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74551749-3516-461d-81d5-8c4e2cd54411-openshift-service-ca-bundle\") pod \"sequence-graph-e8119-7d78b5bfcc-x5zj4\" (UID: \"74551749-3516-461d-81d5-8c4e2cd54411\") " pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:19:16.949276 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.949247    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74551749-3516-461d-81d5-8c4e2cd54411-proxy-tls\") pod \"sequence-graph-e8119-7d78b5bfcc-x5zj4\" (UID: \"74551749-3516-461d-81d5-8c4e2cd54411\") " pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:19:16.949427 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.949282    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74551749-3516-461d-81d5-8c4e2cd54411-openshift-service-ca-bundle\") pod \"sequence-graph-e8119-7d78b5bfcc-x5zj4\" (UID: \"74551749-3516-461d-81d5-8c4e2cd54411\") " pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:19:16.949427 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:19:16.949402    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-e8119-serving-cert: secret "sequence-graph-e8119-serving-cert" not found
Jul 09 18:19:16.949502 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:19:16.949477    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74551749-3516-461d-81d5-8c4e2cd54411-proxy-tls podName:74551749-3516-461d-81d5-8c4e2cd54411 nodeName:}" failed. No retries permitted until 2026-07-09 18:19:17.449459361 +0000 UTC m=+911.783323416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/74551749-3516-461d-81d5-8c4e2cd54411-proxy-tls") pod "sequence-graph-e8119-7d78b5bfcc-x5zj4" (UID: "74551749-3516-461d-81d5-8c4e2cd54411") : secret "sequence-graph-e8119-serving-cert" not found
Jul 09 18:19:16.949984 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:16.949965    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74551749-3516-461d-81d5-8c4e2cd54411-openshift-service-ca-bundle\") pod \"sequence-graph-e8119-7d78b5bfcc-x5zj4\" (UID: \"74551749-3516-461d-81d5-8c4e2cd54411\") " pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:19:17.453556 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:17.453521    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74551749-3516-461d-81d5-8c4e2cd54411-proxy-tls\") pod \"sequence-graph-e8119-7d78b5bfcc-x5zj4\" (UID: \"74551749-3516-461d-81d5-8c4e2cd54411\") " pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:19:17.455845 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:17.455817    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74551749-3516-461d-81d5-8c4e2cd54411-proxy-tls\") pod \"sequence-graph-e8119-7d78b5bfcc-x5zj4\" (UID: \"74551749-3516-461d-81d5-8c4e2cd54411\") " pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:19:17.739788 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:17.739693    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:19:17.866702 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:17.866667    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"]
Jul 09 18:19:17.869706 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:19:17.869677    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74551749_3516_461d_81d5_8c4e2cd54411.slice/crio-2ed3425f217aa693ee99af6f2992dea04ff3e46a3faf5b1a6f80603b9eeafea1 WatchSource:0}: Error finding container 2ed3425f217aa693ee99af6f2992dea04ff3e46a3faf5b1a6f80603b9eeafea1: Status 404 returned error can't find the container with id 2ed3425f217aa693ee99af6f2992dea04ff3e46a3faf5b1a6f80603b9eeafea1
Jul 09 18:19:18.344062 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:18.344023    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" event={"ID":"74551749-3516-461d-81d5-8c4e2cd54411","Type":"ContainerStarted","Data":"3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc"}
Jul 09 18:19:18.344228 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:18.344082    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" event={"ID":"74551749-3516-461d-81d5-8c4e2cd54411","Type":"ContainerStarted","Data":"2ed3425f217aa693ee99af6f2992dea04ff3e46a3faf5b1a6f80603b9eeafea1"}
Jul 09 18:19:18.344228 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:18.344114    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:19:18.361163 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:18.361112    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" podStartSLOduration=2.361097039 podStartE2EDuration="2.361097039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:19:18.359850808 +0000 UTC m=+912.693714886" watchObservedRunningTime="2026-07-09 18:19:18.361097039 +0000 UTC m=+912.694961117"
Jul 09 18:19:24.353493 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:19:24.353462    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:24:06.171710 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:24:06.171633    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:24:06.173085 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:24:06.173067    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:26:55.513519 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:26:55.513446    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"]
Jul 09 18:26:55.515856 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:26:55.513644    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" podUID="04243aba-085d-469a-938c-db9675cc3ab5" containerName="switch-graph-2a7bf" containerID="cri-o://58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645" gracePeriod=30
Jul 09 18:26:58.223128 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:26:58.223078    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" podUID="04243aba-085d-469a-938c-db9675cc3ab5" containerName="switch-graph-2a7bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:27:03.222981 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:03.222941    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" podUID="04243aba-085d-469a-938c-db9675cc3ab5" containerName="switch-graph-2a7bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:27:08.222572 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:08.222536    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" podUID="04243aba-085d-469a-938c-db9675cc3ab5" containerName="switch-graph-2a7bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:27:08.225793 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:08.225770    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:27:13.223216 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:13.223169    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" podUID="04243aba-085d-469a-938c-db9675cc3ab5" containerName="switch-graph-2a7bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:27:18.222970 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:18.222930    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" podUID="04243aba-085d-469a-938c-db9675cc3ab5" containerName="switch-graph-2a7bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:27:23.223126 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:23.223077    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" podUID="04243aba-085d-469a-938c-db9675cc3ab5" containerName="switch-graph-2a7bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:27:25.662700 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:25.662670    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:27:25.731804 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:25.731773    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04243aba-085d-469a-938c-db9675cc3ab5-openshift-service-ca-bundle\") pod \"04243aba-085d-469a-938c-db9675cc3ab5\" (UID: \"04243aba-085d-469a-938c-db9675cc3ab5\") "
Jul 09 18:27:25.731981 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:25.731814    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04243aba-085d-469a-938c-db9675cc3ab5-proxy-tls\") pod \"04243aba-085d-469a-938c-db9675cc3ab5\" (UID: \"04243aba-085d-469a-938c-db9675cc3ab5\") "
Jul 09 18:27:25.732207 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:25.732180    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04243aba-085d-469a-938c-db9675cc3ab5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "04243aba-085d-469a-938c-db9675cc3ab5" (UID: "04243aba-085d-469a-938c-db9675cc3ab5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:27:25.733767 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:25.733748    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04243aba-085d-469a-938c-db9675cc3ab5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04243aba-085d-469a-938c-db9675cc3ab5" (UID: "04243aba-085d-469a-938c-db9675cc3ab5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:27:25.833067 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:25.832987    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04243aba-085d-469a-938c-db9675cc3ab5-openshift-service-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:27:25.833067 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:25.833016    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04243aba-085d-469a-938c-db9675cc3ab5-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:27:26.039094 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:26.039060    2568 generic.go:356] "Generic (PLEG): container finished" podID="04243aba-085d-469a-938c-db9675cc3ab5" containerID="58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645" exitCode=0
Jul 09 18:27:26.039274 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:26.039102    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"
Jul 09 18:27:26.039274 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:26.039133    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" event={"ID":"04243aba-085d-469a-938c-db9675cc3ab5","Type":"ContainerDied","Data":"58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645"}
Jul 09 18:27:26.039274 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:26.039213    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj" event={"ID":"04243aba-085d-469a-938c-db9675cc3ab5","Type":"ContainerDied","Data":"848cdd6b08bf11c77daca0e8e7670a7d755213a17528422b0e4f2c0e1fcb9f51"}
Jul 09 18:27:26.039274 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:26.039244    2568 scope.go:117] "RemoveContainer" containerID="58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645"
Jul 09 18:27:26.048467 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:26.048449    2568 scope.go:117] "RemoveContainer" containerID="58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645"
Jul 09 18:27:26.048742 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:27:26.048722    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645\": container with ID starting with 58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645 not found: ID does not exist" containerID="58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645"
Jul 09 18:27:26.048817 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:26.048762    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645"} err="failed to get container status \"58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645\": rpc error: code = NotFound desc = could not find container \"58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645\": container with ID starting with 58fe2b216455b42343baaeb3d08c1a7df2077b74425cde4bc1e79873acdab645 not found: ID does not exist"
Jul 09 18:27:26.065170 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:26.065150    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"]
Jul 09 18:27:26.068428 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:26.068405    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-2a7bf-59d54ccb5d-7s4cj"]
Jul 09 18:27:26.226191 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:26.226112    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04243aba-085d-469a-938c-db9675cc3ab5" path="/var/lib/kubelet/pods/04243aba-085d-469a-938c-db9675cc3ab5/volumes"
Jul 09 18:27:31.566334 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:31.566289    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"]
Jul 09 18:27:31.566859 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:31.566549    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" podUID="74551749-3516-461d-81d5-8c4e2cd54411" containerName="sequence-graph-e8119" containerID="cri-o://3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc" gracePeriod=30
Jul 09 18:27:34.351425 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:34.351389    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" podUID="74551749-3516-461d-81d5-8c4e2cd54411" containerName="sequence-graph-e8119" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:27:39.351642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:39.351606    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" podUID="74551749-3516-461d-81d5-8c4e2cd54411" containerName="sequence-graph-e8119" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:27:44.351901 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:44.351845    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" podUID="74551749-3516-461d-81d5-8c4e2cd54411" containerName="sequence-graph-e8119" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:27:44.352342 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:44.351984    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:27:49.351490 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:49.351446    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" podUID="74551749-3516-461d-81d5-8c4e2cd54411" containerName="sequence-graph-e8119" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:27:54.351707 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:54.351666    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" podUID="74551749-3516-461d-81d5-8c4e2cd54411" containerName="sequence-graph-e8119" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:27:55.810286 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.810250    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"]
Jul 09 18:27:55.810676 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.810556    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04243aba-085d-469a-938c-db9675cc3ab5" containerName="switch-graph-2a7bf"
Jul 09 18:27:55.810676 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.810569    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="04243aba-085d-469a-938c-db9675cc3ab5" containerName="switch-graph-2a7bf"
Jul 09 18:27:55.810676 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.810615    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="04243aba-085d-469a-938c-db9675cc3ab5" containerName="switch-graph-2a7bf"
Jul 09 18:27:55.813385 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.813364    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:27:55.816506 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.816488    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-64fa7-kube-rbac-proxy-sar-config\""
Jul 09 18:27:55.816613 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.816581    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-64fa7-serving-cert\""
Jul 09 18:27:55.828375 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.828354    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"]
Jul 09 18:27:55.839535 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.839510    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbaa24f3-268d-4bea-8f48-e8002a2f5449-proxy-tls\") pod \"ensemble-graph-64fa7-78bfdfc7d7-vhxjl\" (UID: \"dbaa24f3-268d-4bea-8f48-e8002a2f5449\") " pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:27:55.839625 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.839557    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbaa24f3-268d-4bea-8f48-e8002a2f5449-openshift-service-ca-bundle\") pod \"ensemble-graph-64fa7-78bfdfc7d7-vhxjl\" (UID: \"dbaa24f3-268d-4bea-8f48-e8002a2f5449\") " pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:27:55.940462 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.940428    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbaa24f3-268d-4bea-8f48-e8002a2f5449-openshift-service-ca-bundle\") pod \"ensemble-graph-64fa7-78bfdfc7d7-vhxjl\" (UID: \"dbaa24f3-268d-4bea-8f48-e8002a2f5449\") " pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:27:55.940638 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.940485    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbaa24f3-268d-4bea-8f48-e8002a2f5449-proxy-tls\") pod \"ensemble-graph-64fa7-78bfdfc7d7-vhxjl\" (UID: \"dbaa24f3-268d-4bea-8f48-e8002a2f5449\") " pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:27:55.940638 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:27:55.940598    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-64fa7-serving-cert: secret "ensemble-graph-64fa7-serving-cert" not found
Jul 09 18:27:55.940710 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:27:55.940655    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbaa24f3-268d-4bea-8f48-e8002a2f5449-proxy-tls podName:dbaa24f3-268d-4bea-8f48-e8002a2f5449 nodeName:}" failed. No retries permitted until 2026-07-09 18:27:56.440635082 +0000 UTC m=+1430.774499149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dbaa24f3-268d-4bea-8f48-e8002a2f5449-proxy-tls") pod "ensemble-graph-64fa7-78bfdfc7d7-vhxjl" (UID: "dbaa24f3-268d-4bea-8f48-e8002a2f5449") : secret "ensemble-graph-64fa7-serving-cert" not found
Jul 09 18:27:55.941095 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:55.941072    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbaa24f3-268d-4bea-8f48-e8002a2f5449-openshift-service-ca-bundle\") pod \"ensemble-graph-64fa7-78bfdfc7d7-vhxjl\" (UID: \"dbaa24f3-268d-4bea-8f48-e8002a2f5449\") " pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:27:56.446018 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:56.445981    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbaa24f3-268d-4bea-8f48-e8002a2f5449-proxy-tls\") pod \"ensemble-graph-64fa7-78bfdfc7d7-vhxjl\" (UID: \"dbaa24f3-268d-4bea-8f48-e8002a2f5449\") " pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:27:56.448456 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:56.448431    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbaa24f3-268d-4bea-8f48-e8002a2f5449-proxy-tls\") pod \"ensemble-graph-64fa7-78bfdfc7d7-vhxjl\" (UID: \"dbaa24f3-268d-4bea-8f48-e8002a2f5449\") " pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:27:56.725672 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:56.725587    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:27:56.855510 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:56.855486    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"]
Jul 09 18:27:56.858154 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:27:56.858111    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbaa24f3_268d_4bea_8f48_e8002a2f5449.slice/crio-898bc76547133f96197f3813b7cc7c44a4dbe7c0e04556f08a6d067d5c0aecfd WatchSource:0}: Error finding container 898bc76547133f96197f3813b7cc7c44a4dbe7c0e04556f08a6d067d5c0aecfd: Status 404 returned error can't find the container with id 898bc76547133f96197f3813b7cc7c44a4dbe7c0e04556f08a6d067d5c0aecfd
Jul 09 18:27:57.157273 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:57.157240    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" event={"ID":"dbaa24f3-268d-4bea-8f48-e8002a2f5449","Type":"ContainerStarted","Data":"5ffbb0c25200fa1f32556f66f1a3010a442226e525948449a88e9b2b36c025b5"}
Jul 09 18:27:57.157469 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:57.157300    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" event={"ID":"dbaa24f3-268d-4bea-8f48-e8002a2f5449","Type":"ContainerStarted","Data":"898bc76547133f96197f3813b7cc7c44a4dbe7c0e04556f08a6d067d5c0aecfd"}
Jul 09 18:27:57.157469 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:57.157326    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:27:57.174205 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:57.174157    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" podStartSLOduration=2.174140418 podStartE2EDuration="2.174140418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:27:57.17320602 +0000 UTC m=+1431.507070122" watchObservedRunningTime="2026-07-09 18:27:57.174140418 +0000 UTC m=+1431.508004495"
Jul 09 18:27:59.351927 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:27:59.351831    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" podUID="74551749-3516-461d-81d5-8c4e2cd54411" containerName="sequence-graph-e8119" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:28:01.714868 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:01.714844    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:28:01.787365 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:01.787337    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74551749-3516-461d-81d5-8c4e2cd54411-openshift-service-ca-bundle\") pod \"74551749-3516-461d-81d5-8c4e2cd54411\" (UID: \"74551749-3516-461d-81d5-8c4e2cd54411\") "
Jul 09 18:28:01.787522 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:01.787394    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74551749-3516-461d-81d5-8c4e2cd54411-proxy-tls\") pod \"74551749-3516-461d-81d5-8c4e2cd54411\" (UID: \"74551749-3516-461d-81d5-8c4e2cd54411\") "
Jul 09 18:28:01.787726 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:01.787699    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74551749-3516-461d-81d5-8c4e2cd54411-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "74551749-3516-461d-81d5-8c4e2cd54411" (UID: "74551749-3516-461d-81d5-8c4e2cd54411"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:01.789375 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:01.789358    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74551749-3516-461d-81d5-8c4e2cd54411-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74551749-3516-461d-81d5-8c4e2cd54411" (UID: "74551749-3516-461d-81d5-8c4e2cd54411"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:01.888435 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:01.888355    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74551749-3516-461d-81d5-8c4e2cd54411-openshift-service-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:28:01.888435 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:01.888384    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74551749-3516-461d-81d5-8c4e2cd54411-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:28:02.176743 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:02.176650    2568 generic.go:356] "Generic (PLEG): container finished" podID="74551749-3516-461d-81d5-8c4e2cd54411" containerID="3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc" exitCode=0
Jul 09 18:28:02.176743 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:02.176703    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"
Jul 09 18:28:02.176743 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:02.176732    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" event={"ID":"74551749-3516-461d-81d5-8c4e2cd54411","Type":"ContainerDied","Data":"3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc"}
Jul 09 18:28:02.177003 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:02.176805    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4" event={"ID":"74551749-3516-461d-81d5-8c4e2cd54411","Type":"ContainerDied","Data":"2ed3425f217aa693ee99af6f2992dea04ff3e46a3faf5b1a6f80603b9eeafea1"}
Jul 09 18:28:02.177003 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:02.176828    2568 scope.go:117] "RemoveContainer" containerID="3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc"
Jul 09 18:28:02.185282 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:02.185268    2568 scope.go:117] "RemoveContainer" containerID="3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc"
Jul 09 18:28:02.185514 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:28:02.185497    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc\": container with ID starting with 3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc not found: ID does not exist" containerID="3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc"
Jul 09 18:28:02.185556 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:02.185521    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc"} err="failed to get container status \"3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc\": rpc error: code = NotFound desc = could not find container \"3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc\": container with ID starting with 3bdf4b816069cdc39111833e339c9e3d7b52cbf261fc6bb799aac2cbc5b858bc not found: ID does not exist"
Jul 09 18:28:02.206024 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:02.205997    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"]
Jul 09 18:28:02.212162 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:02.212144    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-e8119-7d78b5bfcc-x5zj4"]
Jul 09 18:28:02.226356 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:02.226329    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74551749-3516-461d-81d5-8c4e2cd54411" path="/var/lib/kubelet/pods/74551749-3516-461d-81d5-8c4e2cd54411/volumes"
Jul 09 18:28:03.166415 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:03.166382    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:28:05.903091 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:05.903063    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"]
Jul 09 18:28:05.903459 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:05.903233    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" podUID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" containerName="ensemble-graph-64fa7" containerID="cri-o://5ffbb0c25200fa1f32556f66f1a3010a442226e525948449a88e9b2b36c025b5" gracePeriod=30
Jul 09 18:28:08.164966 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:08.164929    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" podUID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" containerName="ensemble-graph-64fa7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:28:13.164452 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:13.164415    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" podUID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" containerName="ensemble-graph-64fa7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:28:18.165273 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:18.165234    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" podUID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" containerName="ensemble-graph-64fa7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:28:18.165647 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:18.165339    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:28:23.164846 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:23.164805    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" podUID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" containerName="ensemble-graph-64fa7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:28:28.164332 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:28.164286    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" podUID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" containerName="ensemble-graph-64fa7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:28:31.833631 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:31.833601    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"]
Jul 09 18:28:31.833999 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:31.833901    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74551749-3516-461d-81d5-8c4e2cd54411" containerName="sequence-graph-e8119"
Jul 09 18:28:31.833999 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:31.833915    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="74551749-3516-461d-81d5-8c4e2cd54411" containerName="sequence-graph-e8119"
Jul 09 18:28:31.833999 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:31.833972    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="74551749-3516-461d-81d5-8c4e2cd54411" containerName="sequence-graph-e8119"
Jul 09 18:28:31.838195 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:31.838178    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:28:31.841242 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:31.841222    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4dc46-serving-cert\""
Jul 09 18:28:31.841650 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:31.841631    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4dc46-kube-rbac-proxy-sar-config\""
Jul 09 18:28:31.850692 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:31.850668    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"]
Jul 09 18:28:31.917643 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:31.917613    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b82f720-ead5-4008-9266-af7b58bb6683-openshift-service-ca-bundle\") pod \"sequence-graph-4dc46-c85964bb8-9gb4g\" (UID: \"2b82f720-ead5-4008-9266-af7b58bb6683\") " pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:28:31.917643 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:31.917644    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b82f720-ead5-4008-9266-af7b58bb6683-proxy-tls\") pod \"sequence-graph-4dc46-c85964bb8-9gb4g\" (UID: \"2b82f720-ead5-4008-9266-af7b58bb6683\") " pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:28:32.018651 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:32.018616    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b82f720-ead5-4008-9266-af7b58bb6683-openshift-service-ca-bundle\") pod \"sequence-graph-4dc46-c85964bb8-9gb4g\" (UID: \"2b82f720-ead5-4008-9266-af7b58bb6683\") " pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:28:32.018651 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:32.018653    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b82f720-ead5-4008-9266-af7b58bb6683-proxy-tls\") pod \"sequence-graph-4dc46-c85964bb8-9gb4g\" (UID: \"2b82f720-ead5-4008-9266-af7b58bb6683\") " pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:28:32.019274 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:32.019248    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b82f720-ead5-4008-9266-af7b58bb6683-openshift-service-ca-bundle\") pod \"sequence-graph-4dc46-c85964bb8-9gb4g\" (UID: \"2b82f720-ead5-4008-9266-af7b58bb6683\") " pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:28:32.021131 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:32.021096    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b82f720-ead5-4008-9266-af7b58bb6683-proxy-tls\") pod \"sequence-graph-4dc46-c85964bb8-9gb4g\" (UID: \"2b82f720-ead5-4008-9266-af7b58bb6683\") " pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:28:32.148639 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:32.148543    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:28:32.294201 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:32.294180    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"]
Jul 09 18:28:32.296450 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:28:32.296426    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b82f720_ead5_4008_9266_af7b58bb6683.slice/crio-5a82931b7c345d04cda304204e1cceefc1c2ff2a7193508cde13f0a63028837a WatchSource:0}: Error finding container 5a82931b7c345d04cda304204e1cceefc1c2ff2a7193508cde13f0a63028837a: Status 404 returned error can't find the container with id 5a82931b7c345d04cda304204e1cceefc1c2ff2a7193508cde13f0a63028837a
Jul 09 18:28:33.164900 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:33.164859    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" podUID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" containerName="ensemble-graph-64fa7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:28:33.288851 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:33.288818    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" event={"ID":"2b82f720-ead5-4008-9266-af7b58bb6683","Type":"ContainerStarted","Data":"41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5"}
Jul 09 18:28:33.289022 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:33.288868    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" event={"ID":"2b82f720-ead5-4008-9266-af7b58bb6683","Type":"ContainerStarted","Data":"5a82931b7c345d04cda304204e1cceefc1c2ff2a7193508cde13f0a63028837a"}
Jul 09 18:28:33.289022 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:33.289012    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:28:33.308987 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:33.308947    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" podStartSLOduration=2.3089310530000002 podStartE2EDuration="2.308931053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:33.307155581 +0000 UTC m=+1467.641019684" watchObservedRunningTime="2026-07-09 18:28:33.308931053 +0000 UTC m=+1467.642795131"
Jul 09 18:28:36.301088 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:36.301056    2568 generic.go:356] "Generic (PLEG): container finished" podID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" containerID="5ffbb0c25200fa1f32556f66f1a3010a442226e525948449a88e9b2b36c025b5" exitCode=0
Jul 09 18:28:36.301472 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:36.301126    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" event={"ID":"dbaa24f3-268d-4bea-8f48-e8002a2f5449","Type":"ContainerDied","Data":"5ffbb0c25200fa1f32556f66f1a3010a442226e525948449a88e9b2b36c025b5"}
Jul 09 18:28:36.544172 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:36.544145    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:28:36.652324 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:36.652243    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbaa24f3-268d-4bea-8f48-e8002a2f5449-proxy-tls\") pod \"dbaa24f3-268d-4bea-8f48-e8002a2f5449\" (UID: \"dbaa24f3-268d-4bea-8f48-e8002a2f5449\") "
Jul 09 18:28:36.652484 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:36.652332    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbaa24f3-268d-4bea-8f48-e8002a2f5449-openshift-service-ca-bundle\") pod \"dbaa24f3-268d-4bea-8f48-e8002a2f5449\" (UID: \"dbaa24f3-268d-4bea-8f48-e8002a2f5449\") "
Jul 09 18:28:36.652698 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:36.652669    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbaa24f3-268d-4bea-8f48-e8002a2f5449-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "dbaa24f3-268d-4bea-8f48-e8002a2f5449" (UID: "dbaa24f3-268d-4bea-8f48-e8002a2f5449"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:36.654216 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:36.654195    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbaa24f3-268d-4bea-8f48-e8002a2f5449-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbaa24f3-268d-4bea-8f48-e8002a2f5449" (UID: "dbaa24f3-268d-4bea-8f48-e8002a2f5449"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:36.753250 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:36.753214    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbaa24f3-268d-4bea-8f48-e8002a2f5449-openshift-service-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:28:36.753250 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:36.753245    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbaa24f3-268d-4bea-8f48-e8002a2f5449-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:28:37.305578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:37.305548    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"
Jul 09 18:28:37.305578 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:37.305569    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl" event={"ID":"dbaa24f3-268d-4bea-8f48-e8002a2f5449","Type":"ContainerDied","Data":"898bc76547133f96197f3813b7cc7c44a4dbe7c0e04556f08a6d067d5c0aecfd"}
Jul 09 18:28:37.306132 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:37.305617    2568 scope.go:117] "RemoveContainer" containerID="5ffbb0c25200fa1f32556f66f1a3010a442226e525948449a88e9b2b36c025b5"
Jul 09 18:28:37.330611 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:37.330588    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"]
Jul 09 18:28:37.336476 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:37.336456    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-64fa7-78bfdfc7d7-vhxjl"]
Jul 09 18:28:38.225689 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:38.225646    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" path="/var/lib/kubelet/pods/dbaa24f3-268d-4bea-8f48-e8002a2f5449/volumes"
Jul 09 18:28:39.299672 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:39.299636    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:28:41.950180 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:41.950137    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"]
Jul 09 18:28:41.950622 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:41.950361    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" podUID="2b82f720-ead5-4008-9266-af7b58bb6683" containerName="sequence-graph-4dc46" containerID="cri-o://41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5" gracePeriod=30
Jul 09 18:28:44.297469 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:44.297428    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" podUID="2b82f720-ead5-4008-9266-af7b58bb6683" containerName="sequence-graph-4dc46" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:28:49.296730 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:49.296693    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" podUID="2b82f720-ead5-4008-9266-af7b58bb6683" containerName="sequence-graph-4dc46" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:28:54.297204 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:54.297170    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" podUID="2b82f720-ead5-4008-9266-af7b58bb6683" containerName="sequence-graph-4dc46" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:28:54.297643 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:54.297266    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:28:59.297241 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:28:59.297199    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" podUID="2b82f720-ead5-4008-9266-af7b58bb6683" containerName="sequence-graph-4dc46" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:29:04.296994 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:04.296953    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" podUID="2b82f720-ead5-4008-9266-af7b58bb6683" containerName="sequence-graph-4dc46" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:29:06.167628 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.167594    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"]
Jul 09 18:29:06.167994 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.167929    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" containerName="ensemble-graph-64fa7"
Jul 09 18:29:06.167994 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.167944    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" containerName="ensemble-graph-64fa7"
Jul 09 18:29:06.168067 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.167997    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbaa24f3-268d-4bea-8f48-e8002a2f5449" containerName="ensemble-graph-64fa7"
Jul 09 18:29:06.170671 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.170652    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:29:06.174864 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.174843    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-f76a1-serving-cert\""
Jul 09 18:29:06.174975 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.174876    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-f76a1-kube-rbac-proxy-sar-config\""
Jul 09 18:29:06.177621 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.177584    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-proxy-tls\") pod \"ensemble-graph-f76a1-858bfd4cc6-scmt5\" (UID: \"5856fc3e-81f1-4a8b-a650-7d83bcc13d88\") " pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:29:06.177737 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.177633    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-openshift-service-ca-bundle\") pod \"ensemble-graph-f76a1-858bfd4cc6-scmt5\" (UID: \"5856fc3e-81f1-4a8b-a650-7d83bcc13d88\") " pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:29:06.183285 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.183261    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"]
Jul 09 18:29:06.195818 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.195796    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:29:06.202817 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.202795    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:29:06.278144 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.278114    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-proxy-tls\") pod \"ensemble-graph-f76a1-858bfd4cc6-scmt5\" (UID: \"5856fc3e-81f1-4a8b-a650-7d83bcc13d88\") " pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:29:06.278255 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.278149    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-openshift-service-ca-bundle\") pod \"ensemble-graph-f76a1-858bfd4cc6-scmt5\" (UID: \"5856fc3e-81f1-4a8b-a650-7d83bcc13d88\") " pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:29:06.278738 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.278713    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-openshift-service-ca-bundle\") pod \"ensemble-graph-f76a1-858bfd4cc6-scmt5\" (UID: \"5856fc3e-81f1-4a8b-a650-7d83bcc13d88\") " pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:29:06.280495 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.280474    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-proxy-tls\") pod \"ensemble-graph-f76a1-858bfd4cc6-scmt5\" (UID: \"5856fc3e-81f1-4a8b-a650-7d83bcc13d88\") " pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:29:06.481268 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.481192    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:29:06.612845 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:06.612709    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"]
Jul 09 18:29:06.615414 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:29:06.615387    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5856fc3e_81f1_4a8b_a650_7d83bcc13d88.slice/crio-401373453a38a828588e58847d57159af2e3d30518bb863dc83cb1e5b72e5709 WatchSource:0}: Error finding container 401373453a38a828588e58847d57159af2e3d30518bb863dc83cb1e5b72e5709: Status 404 returned error can't find the container with id 401373453a38a828588e58847d57159af2e3d30518bb863dc83cb1e5b72e5709
Jul 09 18:29:07.419358 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:07.419322    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" event={"ID":"5856fc3e-81f1-4a8b-a650-7d83bcc13d88","Type":"ContainerStarted","Data":"e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135"}
Jul 09 18:29:07.419814 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:07.419376    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" event={"ID":"5856fc3e-81f1-4a8b-a650-7d83bcc13d88","Type":"ContainerStarted","Data":"401373453a38a828588e58847d57159af2e3d30518bb863dc83cb1e5b72e5709"}
Jul 09 18:29:07.419814 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:07.419444    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:29:07.437927 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:07.437855    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" podStartSLOduration=1.437834855 podStartE2EDuration="1.437834855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:29:07.437441833 +0000 UTC m=+1501.771305977" watchObservedRunningTime="2026-07-09 18:29:07.437834855 +0000 UTC m=+1501.771698935"
Jul 09 18:29:09.297589 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:09.297546    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" podUID="2b82f720-ead5-4008-9266-af7b58bb6683" containerName="sequence-graph-4dc46" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:29:12.097928 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.097903    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:29:12.123020 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.122993    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b82f720-ead5-4008-9266-af7b58bb6683-openshift-service-ca-bundle\") pod \"2b82f720-ead5-4008-9266-af7b58bb6683\" (UID: \"2b82f720-ead5-4008-9266-af7b58bb6683\") "
Jul 09 18:29:12.123168 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.123156    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b82f720-ead5-4008-9266-af7b58bb6683-proxy-tls\") pod \"2b82f720-ead5-4008-9266-af7b58bb6683\" (UID: \"2b82f720-ead5-4008-9266-af7b58bb6683\") "
Jul 09 18:29:12.123296 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.123275    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b82f720-ead5-4008-9266-af7b58bb6683-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2b82f720-ead5-4008-9266-af7b58bb6683" (UID: "2b82f720-ead5-4008-9266-af7b58bb6683"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:29:12.125167 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.125134    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b82f720-ead5-4008-9266-af7b58bb6683-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b82f720-ead5-4008-9266-af7b58bb6683" (UID: "2b82f720-ead5-4008-9266-af7b58bb6683"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:29:12.223631 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.223552    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b82f720-ead5-4008-9266-af7b58bb6683-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:29:12.223631 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.223580    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b82f720-ead5-4008-9266-af7b58bb6683-openshift-service-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:29:12.443141 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.443095    2568 generic.go:356] "Generic (PLEG): container finished" podID="2b82f720-ead5-4008-9266-af7b58bb6683" containerID="41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5" exitCode=0
Jul 09 18:29:12.443308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.443188    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"
Jul 09 18:29:12.443308 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.443187    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" event={"ID":"2b82f720-ead5-4008-9266-af7b58bb6683","Type":"ContainerDied","Data":"41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5"}
Jul 09 18:29:12.443389 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.443325    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g" event={"ID":"2b82f720-ead5-4008-9266-af7b58bb6683","Type":"ContainerDied","Data":"5a82931b7c345d04cda304204e1cceefc1c2ff2a7193508cde13f0a63028837a"}
Jul 09 18:29:12.443389 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.443374    2568 scope.go:117] "RemoveContainer" containerID="41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5"
Jul 09 18:29:12.451285 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.451259    2568 scope.go:117] "RemoveContainer" containerID="41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5"
Jul 09 18:29:12.451536 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:29:12.451510    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5\": container with ID starting with 41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5 not found: ID does not exist" containerID="41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5"
Jul 09 18:29:12.451597 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.451541    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5"} err="failed to get container status \"41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5\": rpc error: code = NotFound desc = could not find container \"41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5\": container with ID starting with 41afddf63939c9a4b589b0ed243cd0c55e0858c81d3165a9edc32eac9a97aed5 not found: ID does not exist"
Jul 09 18:29:12.467859 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.467836    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"]
Jul 09 18:29:12.473155 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:12.473138    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4dc46-c85964bb8-9gb4g"]
Jul 09 18:29:13.428332 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:13.428303    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:29:14.226648 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:14.226620    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b82f720-ead5-4008-9266-af7b58bb6683" path="/var/lib/kubelet/pods/2b82f720-ead5-4008-9266-af7b58bb6683/volumes"
Jul 09 18:29:42.242104 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.242022    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"]
Jul 09 18:29:42.242544 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.242312    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b82f720-ead5-4008-9266-af7b58bb6683" containerName="sequence-graph-4dc46"
Jul 09 18:29:42.242544 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.242324    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b82f720-ead5-4008-9266-af7b58bb6683" containerName="sequence-graph-4dc46"
Jul 09 18:29:42.242544 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.242384    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b82f720-ead5-4008-9266-af7b58bb6683" containerName="sequence-graph-4dc46"
Jul 09 18:29:42.245318 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.245297    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:29:42.247877 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.247856    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2ff98-serving-cert\""
Jul 09 18:29:42.248138 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.248118    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2ff98-kube-rbac-proxy-sar-config\""
Jul 09 18:29:42.256401 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.256382    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"]
Jul 09 18:29:42.344667 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.344633    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c647761-30e4-4599-9112-33929a0d054e-proxy-tls\") pod \"sequence-graph-2ff98-56c4fc6bd7-wmkxh\" (UID: \"6c647761-30e4-4599-9112-33929a0d054e\") " pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:29:42.344854 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.344669    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c647761-30e4-4599-9112-33929a0d054e-openshift-service-ca-bundle\") pod \"sequence-graph-2ff98-56c4fc6bd7-wmkxh\" (UID: \"6c647761-30e4-4599-9112-33929a0d054e\") " pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:29:42.445524 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.445476    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c647761-30e4-4599-9112-33929a0d054e-proxy-tls\") pod \"sequence-graph-2ff98-56c4fc6bd7-wmkxh\" (UID: \"6c647761-30e4-4599-9112-33929a0d054e\") " pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:29:42.445524 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.445528    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c647761-30e4-4599-9112-33929a0d054e-openshift-service-ca-bundle\") pod \"sequence-graph-2ff98-56c4fc6bd7-wmkxh\" (UID: \"6c647761-30e4-4599-9112-33929a0d054e\") " pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:29:42.446191 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.446165    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c647761-30e4-4599-9112-33929a0d054e-openshift-service-ca-bundle\") pod \"sequence-graph-2ff98-56c4fc6bd7-wmkxh\" (UID: \"6c647761-30e4-4599-9112-33929a0d054e\") " pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:29:42.447950 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.447929    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c647761-30e4-4599-9112-33929a0d054e-proxy-tls\") pod \"sequence-graph-2ff98-56c4fc6bd7-wmkxh\" (UID: \"6c647761-30e4-4599-9112-33929a0d054e\") " pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:29:42.555494 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.555468    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:29:42.680265 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:42.680242    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"]
Jul 09 18:29:42.682722 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:29:42.682687    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c647761_30e4_4599_9112_33929a0d054e.slice/crio-127ea57bd7a9594827cb18fffebb4be5f9a6683978663a777f42773cc6c9442c WatchSource:0}: Error finding container 127ea57bd7a9594827cb18fffebb4be5f9a6683978663a777f42773cc6c9442c: Status 404 returned error can't find the container with id 127ea57bd7a9594827cb18fffebb4be5f9a6683978663a777f42773cc6c9442c
Jul 09 18:29:43.560000 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:43.559957    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" event={"ID":"6c647761-30e4-4599-9112-33929a0d054e","Type":"ContainerStarted","Data":"33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f"}
Jul 09 18:29:43.560421 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:43.560020    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" event={"ID":"6c647761-30e4-4599-9112-33929a0d054e","Type":"ContainerStarted","Data":"127ea57bd7a9594827cb18fffebb4be5f9a6683978663a777f42773cc6c9442c"}
Jul 09 18:29:43.560421 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:43.560075    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:29:43.576776 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:43.576726    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" podStartSLOduration=1.576710032 podStartE2EDuration="1.576710032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:29:43.575815054 +0000 UTC m=+1537.909679145" watchObservedRunningTime="2026-07-09 18:29:43.576710032 +0000 UTC m=+1537.910574110"
Jul 09 18:29:49.569359 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:29:49.569325    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:34:06.224428 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:34:06.224397    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:34:06.226019 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:34:06.225995    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:37:20.648479 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:20.648446    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"]
Jul 09 18:37:20.649016 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:20.648646    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" podUID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" containerName="ensemble-graph-f76a1" containerID="cri-o://e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135" gracePeriod=30
Jul 09 18:37:23.426921 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:23.426865    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" podUID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" containerName="ensemble-graph-f76a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:37:28.427442 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:28.427408    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" podUID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" containerName="ensemble-graph-f76a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:37:33.426932 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:33.426868    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" podUID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" containerName="ensemble-graph-f76a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:37:33.427288 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:33.427004    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:37:38.426955 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:38.426919    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" podUID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" containerName="ensemble-graph-f76a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:37:43.427528 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:43.427485    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" podUID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" containerName="ensemble-graph-f76a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:37:48.426992 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:48.426952    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" podUID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" containerName="ensemble-graph-f76a1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:37:50.789600 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:50.789572    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:37:50.874733 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:50.874706    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-proxy-tls\") pod \"5856fc3e-81f1-4a8b-a650-7d83bcc13d88\" (UID: \"5856fc3e-81f1-4a8b-a650-7d83bcc13d88\") "
Jul 09 18:37:50.874910 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:50.874772    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-openshift-service-ca-bundle\") pod \"5856fc3e-81f1-4a8b-a650-7d83bcc13d88\" (UID: \"5856fc3e-81f1-4a8b-a650-7d83bcc13d88\") "
Jul 09 18:37:50.875111 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:50.875084    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5856fc3e-81f1-4a8b-a650-7d83bcc13d88" (UID: "5856fc3e-81f1-4a8b-a650-7d83bcc13d88"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:37:50.876753 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:50.876728    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5856fc3e-81f1-4a8b-a650-7d83bcc13d88" (UID: "5856fc3e-81f1-4a8b-a650-7d83bcc13d88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:37:50.975474 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:50.975398    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-openshift-service-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:37:50.975474 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:50.975424    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5856fc3e-81f1-4a8b-a650-7d83bcc13d88-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:37:51.269090 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:51.269059    2568 generic.go:356] "Generic (PLEG): container finished" podID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" containerID="e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135" exitCode=0
Jul 09 18:37:51.269268 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:51.269102    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"
Jul 09 18:37:51.269268 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:51.269149    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" event={"ID":"5856fc3e-81f1-4a8b-a650-7d83bcc13d88","Type":"ContainerDied","Data":"e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135"}
Jul 09 18:37:51.269268 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:51.269204    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5" event={"ID":"5856fc3e-81f1-4a8b-a650-7d83bcc13d88","Type":"ContainerDied","Data":"401373453a38a828588e58847d57159af2e3d30518bb863dc83cb1e5b72e5709"}
Jul 09 18:37:51.269268 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:51.269232    2568 scope.go:117] "RemoveContainer" containerID="e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135"
Jul 09 18:37:51.277673 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:51.277652    2568 scope.go:117] "RemoveContainer" containerID="e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135"
Jul 09 18:37:51.277925 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:37:51.277907    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135\": container with ID starting with e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135 not found: ID does not exist" containerID="e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135"
Jul 09 18:37:51.277991 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:51.277935    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135"} err="failed to get container status \"e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135\": rpc error: code = NotFound desc = could not find container \"e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135\": container with ID starting with e93f83864a82bd476192c1197dc9dfb795a1e32ddf7d2b01ea7f516ab8a53135 not found: ID does not exist"
Jul 09 18:37:51.291479 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:51.291456    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"]
Jul 09 18:37:51.303845 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:51.303823    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-f76a1-858bfd4cc6-scmt5"]
Jul 09 18:37:52.225412 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:52.225378    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" path="/var/lib/kubelet/pods/5856fc3e-81f1-4a8b-a650-7d83bcc13d88/volumes"
Jul 09 18:37:56.901807 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:56.901774    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"]
Jul 09 18:37:56.902286 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:56.901980    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" podUID="6c647761-30e4-4599-9112-33929a0d054e" containerName="sequence-graph-2ff98" containerID="cri-o://33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f" gracePeriod=30
Jul 09 18:37:59.569002 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:37:59.568959    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" podUID="6c647761-30e4-4599-9112-33929a0d054e" containerName="sequence-graph-2ff98" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:04.568657 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:04.568617    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" podUID="6c647761-30e4-4599-9112-33929a0d054e" containerName="sequence-graph-2ff98" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:09.568501 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:09.568459    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" podUID="6c647761-30e4-4599-9112-33929a0d054e" containerName="sequence-graph-2ff98" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:09.568880 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:09.568562    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:38:14.568479 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:14.568433    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" podUID="6c647761-30e4-4599-9112-33929a0d054e" containerName="sequence-graph-2ff98" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:19.568312 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:19.568266    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" podUID="6c647761-30e4-4599-9112-33929a0d054e" containerName="sequence-graph-2ff98" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:20.944289 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:20.944256    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"]
Jul 09 18:38:20.944678 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:20.944541    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" containerName="ensemble-graph-f76a1"
Jul 09 18:38:20.944678 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:20.944553    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" containerName="ensemble-graph-f76a1"
Jul 09 18:38:20.944678 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:20.944594    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5856fc3e-81f1-4a8b-a650-7d83bcc13d88" containerName="ensemble-graph-f76a1"
Jul 09 18:38:20.947033 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:20.947015    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:38:20.949841 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:20.949814    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-eb3ff-kube-rbac-proxy-sar-config\""
Jul 09 18:38:20.949841 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:20.949814    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-eb3ff-serving-cert\""
Jul 09 18:38:20.955757 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:20.955729    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"]
Jul 09 18:38:21.097350 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:21.097316    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-openshift-service-ca-bundle\") pod \"splitter-graph-eb3ff-7b94c5c474-v8g2q\" (UID: \"f91efcb4-c96f-4ac3-a670-2ce71b1019e5\") " pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:38:21.097503 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:21.097365    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-proxy-tls\") pod \"splitter-graph-eb3ff-7b94c5c474-v8g2q\" (UID: \"f91efcb4-c96f-4ac3-a670-2ce71b1019e5\") " pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:38:21.198550 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:21.198461    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-openshift-service-ca-bundle\") pod \"splitter-graph-eb3ff-7b94c5c474-v8g2q\" (UID: \"f91efcb4-c96f-4ac3-a670-2ce71b1019e5\") " pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:38:21.198550 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:21.198515    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-proxy-tls\") pod \"splitter-graph-eb3ff-7b94c5c474-v8g2q\" (UID: \"f91efcb4-c96f-4ac3-a670-2ce71b1019e5\") " pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:38:21.199151 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:21.199130    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-openshift-service-ca-bundle\") pod \"splitter-graph-eb3ff-7b94c5c474-v8g2q\" (UID: \"f91efcb4-c96f-4ac3-a670-2ce71b1019e5\") " pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:38:21.200844 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:21.200827    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-proxy-tls\") pod \"splitter-graph-eb3ff-7b94c5c474-v8g2q\" (UID: \"f91efcb4-c96f-4ac3-a670-2ce71b1019e5\") " pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:38:21.257573 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:21.257547    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:38:21.385158 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:21.385033    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"]
Jul 09 18:38:21.387190 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:38:21.387161    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91efcb4_c96f_4ac3_a670_2ce71b1019e5.slice/crio-24e22af2ac0390577e40e01188a8d739b7a45b4bdc22acaeb17b4bdfb8a2b13b WatchSource:0}: Error finding container 24e22af2ac0390577e40e01188a8d739b7a45b4bdc22acaeb17b4bdfb8a2b13b: Status 404 returned error can't find the container with id 24e22af2ac0390577e40e01188a8d739b7a45b4bdc22acaeb17b4bdfb8a2b13b
Jul 09 18:38:22.380629 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:22.380598    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" event={"ID":"f91efcb4-c96f-4ac3-a670-2ce71b1019e5","Type":"ContainerStarted","Data":"7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313"}
Jul 09 18:38:22.381082 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:22.380647    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" event={"ID":"f91efcb4-c96f-4ac3-a670-2ce71b1019e5","Type":"ContainerStarted","Data":"24e22af2ac0390577e40e01188a8d739b7a45b4bdc22acaeb17b4bdfb8a2b13b"}
Jul 09 18:38:22.381082 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:22.380714    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:38:22.398966 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:22.398922    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" podStartSLOduration=2.398909574 podStartE2EDuration="2.398909574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:38:22.397826118 +0000 UTC m=+2056.731690209" watchObservedRunningTime="2026-07-09 18:38:22.398909574 +0000 UTC m=+2056.732773643"
Jul 09 18:38:24.568306 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:24.568263    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" podUID="6c647761-30e4-4599-9112-33929a0d054e" containerName="sequence-graph-2ff98" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:27.041197 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.041173    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:38:27.142177 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.142148    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c647761-30e4-4599-9112-33929a0d054e-openshift-service-ca-bundle\") pod \"6c647761-30e4-4599-9112-33929a0d054e\" (UID: \"6c647761-30e4-4599-9112-33929a0d054e\") "
Jul 09 18:38:27.142340 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.142204    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c647761-30e4-4599-9112-33929a0d054e-proxy-tls\") pod \"6c647761-30e4-4599-9112-33929a0d054e\" (UID: \"6c647761-30e4-4599-9112-33929a0d054e\") "
Jul 09 18:38:27.142497 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.142474    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c647761-30e4-4599-9112-33929a0d054e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6c647761-30e4-4599-9112-33929a0d054e" (UID: "6c647761-30e4-4599-9112-33929a0d054e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:38:27.144135 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.144118    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c647761-30e4-4599-9112-33929a0d054e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c647761-30e4-4599-9112-33929a0d054e" (UID: "6c647761-30e4-4599-9112-33929a0d054e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:38:27.243517 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.243438    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c647761-30e4-4599-9112-33929a0d054e-openshift-service-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:38:27.243517 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.243464    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c647761-30e4-4599-9112-33929a0d054e-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:38:27.398696 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.398665    2568 generic.go:356] "Generic (PLEG): container finished" podID="6c647761-30e4-4599-9112-33929a0d054e" containerID="33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f" exitCode=0
Jul 09 18:38:27.398877 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.398714    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"
Jul 09 18:38:27.398877 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.398749    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" event={"ID":"6c647761-30e4-4599-9112-33929a0d054e","Type":"ContainerDied","Data":"33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f"}
Jul 09 18:38:27.398877 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.398800    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh" event={"ID":"6c647761-30e4-4599-9112-33929a0d054e","Type":"ContainerDied","Data":"127ea57bd7a9594827cb18fffebb4be5f9a6683978663a777f42773cc6c9442c"}
Jul 09 18:38:27.398877 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.398823    2568 scope.go:117] "RemoveContainer" containerID="33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f"
Jul 09 18:38:27.406831 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.406813    2568 scope.go:117] "RemoveContainer" containerID="33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f"
Jul 09 18:38:27.407112 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:38:27.407095    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f\": container with ID starting with 33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f not found: ID does not exist" containerID="33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f"
Jul 09 18:38:27.407167 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.407120    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f"} err="failed to get container status \"33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f\": rpc error: code = NotFound desc = could not find container \"33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f\": container with ID starting with 33c6ef1155b70802af987ae9498fbf53be2fa5946ce7574589ab08b53e664a8f not found: ID does not exist"
Jul 09 18:38:27.422804 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.422783    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"]
Jul 09 18:38:27.427732 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:27.427713    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2ff98-56c4fc6bd7-wmkxh"]
Jul 09 18:38:28.226646 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:28.226575    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c647761-30e4-4599-9112-33929a0d054e" path="/var/lib/kubelet/pods/6c647761-30e4-4599-9112-33929a0d054e/volumes"
Jul 09 18:38:28.389719 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:28.389690    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:38:31.105081 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:31.105048    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"]
Jul 09 18:38:31.105533 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:31.105284    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" podUID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" containerName="splitter-graph-eb3ff" containerID="cri-o://7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313" gracePeriod=30
Jul 09 18:38:33.388272 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:33.388234    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" podUID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" containerName="splitter-graph-eb3ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:38.388725 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:38.388687    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" podUID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" containerName="splitter-graph-eb3ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:43.388486 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:43.388443    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" podUID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" containerName="splitter-graph-eb3ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:43.388933 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:43.388562    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:38:48.388724 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:48.388688    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" podUID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" containerName="splitter-graph-eb3ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:53.388393 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:53.388356    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" podUID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" containerName="splitter-graph-eb3ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:57.195706 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.195674    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"]
Jul 09 18:38:57.196089 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.195970    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c647761-30e4-4599-9112-33929a0d054e" containerName="sequence-graph-2ff98"
Jul 09 18:38:57.196089 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.195981    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c647761-30e4-4599-9112-33929a0d054e" containerName="sequence-graph-2ff98"
Jul 09 18:38:57.196089 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.196036    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c647761-30e4-4599-9112-33929a0d054e" containerName="sequence-graph-2ff98"
Jul 09 18:38:57.198852 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.198830    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:38:57.202029 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.201999    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bed12-serving-cert\""
Jul 09 18:38:57.202183 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.202167    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bed12-kube-rbac-proxy-sar-config\""
Jul 09 18:38:57.207574 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.207549    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"]
Jul 09 18:38:57.372003 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.371966    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1a61844-b49a-4201-9f66-9d82ce78a5b3-proxy-tls\") pod \"switch-graph-bed12-6b95d5d48d-2cr4g\" (UID: \"c1a61844-b49a-4201-9f66-9d82ce78a5b3\") " pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:38:57.372003 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.372003    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a61844-b49a-4201-9f66-9d82ce78a5b3-openshift-service-ca-bundle\") pod \"switch-graph-bed12-6b95d5d48d-2cr4g\" (UID: \"c1a61844-b49a-4201-9f66-9d82ce78a5b3\") " pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:38:57.472505 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.472436    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1a61844-b49a-4201-9f66-9d82ce78a5b3-proxy-tls\") pod \"switch-graph-bed12-6b95d5d48d-2cr4g\" (UID: \"c1a61844-b49a-4201-9f66-9d82ce78a5b3\") " pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:38:57.472505 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.472470    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a61844-b49a-4201-9f66-9d82ce78a5b3-openshift-service-ca-bundle\") pod \"switch-graph-bed12-6b95d5d48d-2cr4g\" (UID: \"c1a61844-b49a-4201-9f66-9d82ce78a5b3\") " pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:38:57.472649 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:38:57.472587    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-bed12-serving-cert: secret "switch-graph-bed12-serving-cert" not found
Jul 09 18:38:57.472684 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:38:57.472652    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a61844-b49a-4201-9f66-9d82ce78a5b3-proxy-tls podName:c1a61844-b49a-4201-9f66-9d82ce78a5b3 nodeName:}" failed. No retries permitted until 2026-07-09 18:38:57.972635519 +0000 UTC m=+2092.306499578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c1a61844-b49a-4201-9f66-9d82ce78a5b3-proxy-tls") pod "switch-graph-bed12-6b95d5d48d-2cr4g" (UID: "c1a61844-b49a-4201-9f66-9d82ce78a5b3") : secret "switch-graph-bed12-serving-cert" not found
Jul 09 18:38:57.473086 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.473070    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a61844-b49a-4201-9f66-9d82ce78a5b3-openshift-service-ca-bundle\") pod \"switch-graph-bed12-6b95d5d48d-2cr4g\" (UID: \"c1a61844-b49a-4201-9f66-9d82ce78a5b3\") " pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:38:57.976763 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.976729    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1a61844-b49a-4201-9f66-9d82ce78a5b3-proxy-tls\") pod \"switch-graph-bed12-6b95d5d48d-2cr4g\" (UID: \"c1a61844-b49a-4201-9f66-9d82ce78a5b3\") " pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:38:57.979238 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:57.979216    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1a61844-b49a-4201-9f66-9d82ce78a5b3-proxy-tls\") pod \"switch-graph-bed12-6b95d5d48d-2cr4g\" (UID: \"c1a61844-b49a-4201-9f66-9d82ce78a5b3\") " pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:38:58.110061 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:58.110027    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:38:58.236349 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:58.236271    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"]
Jul 09 18:38:58.239472 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:38:58.239426    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a61844_b49a_4201_9f66_9d82ce78a5b3.slice/crio-31f56d020b0aca8f82f1c570498df985f93cb7ce8ad18d8b1580196c7c90398c WatchSource:0}: Error finding container 31f56d020b0aca8f82f1c570498df985f93cb7ce8ad18d8b1580196c7c90398c: Status 404 returned error can't find the container with id 31f56d020b0aca8f82f1c570498df985f93cb7ce8ad18d8b1580196c7c90398c
Jul 09 18:38:58.388190 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:58.388145    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" podUID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" containerName="splitter-graph-eb3ff" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:38:58.509738 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:58.509709    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" event={"ID":"c1a61844-b49a-4201-9f66-9d82ce78a5b3","Type":"ContainerStarted","Data":"045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35"}
Jul 09 18:38:58.509916 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:58.509768    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" event={"ID":"c1a61844-b49a-4201-9f66-9d82ce78a5b3","Type":"ContainerStarted","Data":"31f56d020b0aca8f82f1c570498df985f93cb7ce8ad18d8b1580196c7c90398c"}
Jul 09 18:38:58.509916 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:58.509796    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:38:58.527061 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:38:58.527016    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" podStartSLOduration=1.527001261 podStartE2EDuration="1.527001261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:38:58.525963168 +0000 UTC m=+2092.859827260" watchObservedRunningTime="2026-07-09 18:38:58.527001261 +0000 UTC m=+2092.860865349"
Jul 09 18:39:01.242784 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.242757    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:39:01.401466 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.401382    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-proxy-tls\") pod \"f91efcb4-c96f-4ac3-a670-2ce71b1019e5\" (UID: \"f91efcb4-c96f-4ac3-a670-2ce71b1019e5\") "
Jul 09 18:39:01.401466 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.401444    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-openshift-service-ca-bundle\") pod \"f91efcb4-c96f-4ac3-a670-2ce71b1019e5\" (UID: \"f91efcb4-c96f-4ac3-a670-2ce71b1019e5\") "
Jul 09 18:39:01.401856 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.401829    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f91efcb4-c96f-4ac3-a670-2ce71b1019e5" (UID: "f91efcb4-c96f-4ac3-a670-2ce71b1019e5"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:39:01.403422 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.403403    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f91efcb4-c96f-4ac3-a670-2ce71b1019e5" (UID: "f91efcb4-c96f-4ac3-a670-2ce71b1019e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:39:01.502846 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.502819    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:39:01.502846 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.502846    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91efcb4-c96f-4ac3-a670-2ce71b1019e5-openshift-service-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:39:01.520899 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.520859    2568 generic.go:356] "Generic (PLEG): container finished" podID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" containerID="7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313" exitCode=0
Jul 09 18:39:01.521001 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.520918    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"
Jul 09 18:39:01.521001 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.520952    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" event={"ID":"f91efcb4-c96f-4ac3-a670-2ce71b1019e5","Type":"ContainerDied","Data":"7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313"}
Jul 09 18:39:01.521074 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.521010    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q" event={"ID":"f91efcb4-c96f-4ac3-a670-2ce71b1019e5","Type":"ContainerDied","Data":"24e22af2ac0390577e40e01188a8d739b7a45b4bdc22acaeb17b4bdfb8a2b13b"}
Jul 09 18:39:01.521074 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.521038    2568 scope.go:117] "RemoveContainer" containerID="7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313"
Jul 09 18:39:01.529332 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.529316    2568 scope.go:117] "RemoveContainer" containerID="7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313"
Jul 09 18:39:01.529577 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:39:01.529559    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313\": container with ID starting with 7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313 not found: ID does not exist" containerID="7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313"
Jul 09 18:39:01.529619 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.529585    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313"} err="failed to get container status \"7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313\": rpc error: code = NotFound desc = could not find container \"7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313\": container with ID starting with 7be2e854b4ec1af8de19c69d142f1993afa7fc74f70447e5bb7ba75903567313 not found: ID does not exist"
Jul 09 18:39:01.544837 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.544813    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"]
Jul 09 18:39:01.547482 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:01.547458    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-eb3ff-7b94c5c474-v8g2q"]
Jul 09 18:39:02.226081 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:02.226049    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" path="/var/lib/kubelet/pods/f91efcb4-c96f-4ac3-a670-2ce71b1019e5/volumes"
Jul 09 18:39:04.518825 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:04.518798    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:39:06.246311 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:06.246280    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:39:06.247722 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:06.247700    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:39:31.440656 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.440619    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"]
Jul 09 18:39:31.441056 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.440922    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" containerName="splitter-graph-eb3ff"
Jul 09 18:39:31.441056 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.440935    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" containerName="splitter-graph-eb3ff"
Jul 09 18:39:31.441056 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.441000    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91efcb4-c96f-4ac3-a670-2ce71b1019e5" containerName="splitter-graph-eb3ff"
Jul 09 18:39:31.443735 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.443715    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:39:31.446580 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.446554    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-865f9-serving-cert\""
Jul 09 18:39:31.446703 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.446623    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-865f9-kube-rbac-proxy-sar-config\""
Jul 09 18:39:31.454131 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.454110    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"]
Jul 09 18:39:31.512607 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.512565    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-proxy-tls\") pod \"splitter-graph-865f9-6d65869f75-dwhzt\" (UID: \"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f\") " pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:39:31.512772 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.512670    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-openshift-service-ca-bundle\") pod \"splitter-graph-865f9-6d65869f75-dwhzt\" (UID: \"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f\") " pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:39:31.613757 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.613724    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-proxy-tls\") pod \"splitter-graph-865f9-6d65869f75-dwhzt\" (UID: \"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f\") " pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:39:31.613938 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.613788    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-openshift-service-ca-bundle\") pod \"splitter-graph-865f9-6d65869f75-dwhzt\" (UID: \"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f\") " pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:39:31.613938 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:39:31.613866    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-865f9-serving-cert: secret "splitter-graph-865f9-serving-cert" not found
Jul 09 18:39:31.614020 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:39:31.613959    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-proxy-tls podName:1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f nodeName:}" failed. No retries permitted until 2026-07-09 18:39:32.11393968 +0000 UTC m=+2126.447803736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-proxy-tls") pod "splitter-graph-865f9-6d65869f75-dwhzt" (UID: "1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f") : secret "splitter-graph-865f9-serving-cert" not found
Jul 09 18:39:31.614416 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:31.614390    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-openshift-service-ca-bundle\") pod \"splitter-graph-865f9-6d65869f75-dwhzt\" (UID: \"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f\") " pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:39:32.119014 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:32.118973    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-proxy-tls\") pod \"splitter-graph-865f9-6d65869f75-dwhzt\" (UID: \"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f\") " pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:39:32.121263 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:32.121242    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-proxy-tls\") pod \"splitter-graph-865f9-6d65869f75-dwhzt\" (UID: \"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f\") " pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:39:32.354104 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:32.354058    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:39:32.480146 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:32.480122    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"]
Jul 09 18:39:32.482641 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:39:32.482605    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4ea1bc_0cc6_44cd_a5af_ff21f896a42f.slice/crio-221a452536e29edaea209b40469dc43b898766684e4e06f6ca319868c177bb84 WatchSource:0}: Error finding container 221a452536e29edaea209b40469dc43b898766684e4e06f6ca319868c177bb84: Status 404 returned error can't find the container with id 221a452536e29edaea209b40469dc43b898766684e4e06f6ca319868c177bb84
Jul 09 18:39:32.632413 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:32.632375    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" event={"ID":"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f","Type":"ContainerStarted","Data":"48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2"}
Jul 09 18:39:32.632550 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:32.632434    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" event={"ID":"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f","Type":"ContainerStarted","Data":"221a452536e29edaea209b40469dc43b898766684e4e06f6ca319868c177bb84"}
Jul 09 18:39:32.632550 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:32.632492    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:39:32.648620 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:32.648576    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" podStartSLOduration=1.6485631509999998 podStartE2EDuration="1.648563151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:39:32.647855716 +0000 UTC m=+2126.981719795" watchObservedRunningTime="2026-07-09 18:39:32.648563151 +0000 UTC m=+2126.982427229"
Jul 09 18:39:38.641306 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:39:38.641274    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:44:06.267007 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:44:06.266974    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:44:06.269576 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:44:06.269443    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:47:46.072234 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:47:46.072160    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"]
Jul 09 18:47:46.072747 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:47:46.072340    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" podUID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" containerName="splitter-graph-865f9" containerID="cri-o://48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2" gracePeriod=30
Jul 09 18:47:48.640323 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:47:48.640282    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" podUID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" containerName="splitter-graph-865f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:47:53.640196 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:47:53.640153    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" podUID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" containerName="splitter-graph-865f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:47:58.640586 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:47:58.640542    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" podUID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" containerName="splitter-graph-865f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:47:58.641010 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:47:58.640643    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:48:03.640506 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:03.640467    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" podUID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" containerName="splitter-graph-865f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:48:08.640423 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:08.640389    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" podUID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" containerName="splitter-graph-865f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:48:13.640013 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:13.639968    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" podUID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" containerName="splitter-graph-865f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:48:16.210633 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.210609    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:48:16.267039 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.267012    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-proxy-tls\") pod \"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f\" (UID: \"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f\") "
Jul 09 18:48:16.267203 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.267054    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-openshift-service-ca-bundle\") pod \"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f\" (UID: \"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f\") "
Jul 09 18:48:16.267476 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.267451    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" (UID: "1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:48:16.268941 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.268919    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" (UID: "1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:48:16.372635 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.368319    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-openshift-service-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:48:16.372635 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.368373    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:48:16.460357 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.460317    2568 generic.go:356] "Generic (PLEG): container finished" podID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" containerID="48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2" exitCode=0
Jul 09 18:48:16.460357 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.460352    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" event={"ID":"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f","Type":"ContainerDied","Data":"48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2"}
Jul 09 18:48:16.460538 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.460379    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"
Jul 09 18:48:16.460538 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.460432    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt" event={"ID":"1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f","Type":"ContainerDied","Data":"221a452536e29edaea209b40469dc43b898766684e4e06f6ca319868c177bb84"}
Jul 09 18:48:16.460538 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.460457    2568 scope.go:117] "RemoveContainer" containerID="48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2"
Jul 09 18:48:16.470536 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.470515    2568 scope.go:117] "RemoveContainer" containerID="48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2"
Jul 09 18:48:16.470799 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:48:16.470779    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2\": container with ID starting with 48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2 not found: ID does not exist" containerID="48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2"
Jul 09 18:48:16.470946 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.470811    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2"} err="failed to get container status \"48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2\": rpc error: code = NotFound desc = could not find container \"48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2\": container with ID starting with 48bb2119bc92cc385ba913c916ded767a84ae077c7630dd2ed669755392dd1c2 not found: ID does not exist"
Jul 09 18:48:16.481845 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.481823    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"]
Jul 09 18:48:16.485836 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:16.485815    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-865f9-6d65869f75-dwhzt"]
Jul 09 18:48:18.225721 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:48:18.225680    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" path="/var/lib/kubelet/pods/1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f/volumes"
Jul 09 18:49:06.288221 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:49:06.288079    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:49:06.291214 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:49:06.291007    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:54:06.313381 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:54:06.313245    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:54:06.317441 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:54:06.317425    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:55:16.304753 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:16.304673    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"]
Jul 09 18:55:16.305383 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:16.304978    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" podUID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" containerName="switch-graph-bed12" containerID="cri-o://045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35" gracePeriod=30
Jul 09 18:55:19.517861 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:19.517827    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" podUID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" containerName="switch-graph-bed12" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:55:24.517818 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:24.517772    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" podUID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" containerName="switch-graph-bed12" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:55:29.516783 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:29.516739    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" podUID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" containerName="switch-graph-bed12" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:55:29.517167 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:29.516843    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:55:32.927430 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:32.927404    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:33.730078 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:33.730050    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:34.517521 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:34.517477    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" podUID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" containerName="switch-graph-bed12" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:55:34.546504 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:34.546480    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:35.341213 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:35.341179    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:36.134041 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:36.134012    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:36.913725 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:36.913697    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:37.712817 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:37.712788    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:38.528285 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:38.528257    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:39.326778 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:39.326749    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:39.517607 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:39.517569    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" podUID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" containerName="switch-graph-bed12" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:55:40.112582 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:40.112547    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:40.897947 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:40.897919    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:41.682667 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:41.682641    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:42.485083 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:42.485056    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:43.307169 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:43.307143    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bed12-6b95d5d48d-2cr4g_c1a61844-b49a-4201-9f66-9d82ce78a5b3/switch-graph-bed12/0.log"
Jul 09 18:55:44.517853 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:44.517812    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" podUID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" containerName="switch-graph-bed12" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:55:46.444863 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:46.444839    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:55:46.459552 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:46.459529    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a61844-b49a-4201-9f66-9d82ce78a5b3-openshift-service-ca-bundle\") pod \"c1a61844-b49a-4201-9f66-9d82ce78a5b3\" (UID: \"c1a61844-b49a-4201-9f66-9d82ce78a5b3\") "
Jul 09 18:55:46.459837 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:46.459813    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a61844-b49a-4201-9f66-9d82ce78a5b3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c1a61844-b49a-4201-9f66-9d82ce78a5b3" (UID: "c1a61844-b49a-4201-9f66-9d82ce78a5b3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:55:46.560871 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:46.560788    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1a61844-b49a-4201-9f66-9d82ce78a5b3-proxy-tls\") pod \"c1a61844-b49a-4201-9f66-9d82ce78a5b3\" (UID: \"c1a61844-b49a-4201-9f66-9d82ce78a5b3\") "
Jul 09 18:55:46.561055 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:46.561035    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a61844-b49a-4201-9f66-9d82ce78a5b3-openshift-service-ca-bundle\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:55:46.562681 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:46.562657    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a61844-b49a-4201-9f66-9d82ce78a5b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1a61844-b49a-4201-9f66-9d82ce78a5b3" (UID: "c1a61844-b49a-4201-9f66-9d82ce78a5b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:55:46.662031 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:46.662002    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1a61844-b49a-4201-9f66-9d82ce78a5b3-proxy-tls\") on node \"ip-10-0-140-234.ec2.internal\" DevicePath \"\""
Jul 09 18:55:47.023332 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:47.023299    2568 generic.go:356] "Generic (PLEG): container finished" podID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" containerID="045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35" exitCode=0
Jul 09 18:55:47.023602 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:47.023348    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"
Jul 09 18:55:47.023602 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:47.023386    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" event={"ID":"c1a61844-b49a-4201-9f66-9d82ce78a5b3","Type":"ContainerDied","Data":"045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35"}
Jul 09 18:55:47.023602 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:47.023438    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g" event={"ID":"c1a61844-b49a-4201-9f66-9d82ce78a5b3","Type":"ContainerDied","Data":"31f56d020b0aca8f82f1c570498df985f93cb7ce8ad18d8b1580196c7c90398c"}
Jul 09 18:55:47.023602 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:47.023462    2568 scope.go:117] "RemoveContainer" containerID="045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35"
Jul 09 18:55:47.032305 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:47.032289    2568 scope.go:117] "RemoveContainer" containerID="045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35"
Jul 09 18:55:47.032554 ip-10-0-140-234 kubenswrapper[2568]: E0709 18:55:47.032538    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35\": container with ID starting with 045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35 not found: ID does not exist" containerID="045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35"
Jul 09 18:55:47.032617 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:47.032565    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35"} err="failed to get container status \"045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35\": rpc error: code = NotFound desc = could not find container \"045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35\": container with ID starting with 045f694ac14dd0b8d3e0db24ec889980b1aa3ea3414ba4eea9b29a119699ff35 not found: ID does not exist"
Jul 09 18:55:47.044940 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:47.044913    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"]
Jul 09 18:55:47.048840 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:47.048821    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bed12-6b95d5d48d-2cr4g"]
Jul 09 18:55:48.226252 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:48.226218    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" path="/var/lib/kubelet/pods/c1a61844-b49a-4201-9f66-9d82ce78a5b3/volumes"
Jul 09 18:55:50.312210 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:50.312180    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pqd8h_fd84a8bd-6c02-477b-8e99-6b78466bea4a/global-pull-secret-syncer/0.log"
Jul 09 18:55:50.483414 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:50.483384    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xm9lj_087b34f5-8d0f-4612-9f19-a87dd8fbf564/konnectivity-agent/0.log"
Jul 09 18:55:50.603084 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:50.603015    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-234.ec2.internal_27720bce3ffdbdb8ce1870f3c11220d9/haproxy/0.log"
Jul 09 18:55:54.833398 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:54.833369    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-m4rhb_b4c5a686-b844-4583-a2c0-9046a49ccc21/kube-state-metrics/0.log"
Jul 09 18:55:54.854351 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:54.854327    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-m4rhb_b4c5a686-b844-4583-a2c0-9046a49ccc21/kube-rbac-proxy-main/0.log"
Jul 09 18:55:54.876620 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:54.876600    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-m4rhb_b4c5a686-b844-4583-a2c0-9046a49ccc21/kube-rbac-proxy-self/0.log"
Jul 09 18:55:55.117779 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:55.117698    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhn4h_662d0433-b416-49a5-b4f8-0849b5508cc4/node-exporter/0.log"
Jul 09 18:55:55.142237 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:55.142215    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhn4h_662d0433-b416-49a5-b4f8-0849b5508cc4/kube-rbac-proxy/0.log"
Jul 09 18:55:55.168813 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:55.168793    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhn4h_662d0433-b416-49a5-b4f8-0849b5508cc4/init-textfile/0.log"
Jul 09 18:55:58.002781 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.002750    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7f8455744f-n7gvk_8c4910e1-16f9-41df-adca-6381c1685d4d/console/0.log"
Jul 09 18:55:58.686825 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.686790    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"]
Jul 09 18:55:58.687080 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.687067    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" containerName="splitter-graph-865f9"
Jul 09 18:55:58.687134 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.687081    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" containerName="splitter-graph-865f9"
Jul 09 18:55:58.687134 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.687091    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" containerName="switch-graph-bed12"
Jul 09 18:55:58.687134 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.687097    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" containerName="switch-graph-bed12"
Jul 09 18:55:58.687239 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.687146    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a61844-b49a-4201-9f66-9d82ce78a5b3" containerName="switch-graph-bed12"
Jul 09 18:55:58.687239 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.687156    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a4ea1bc-0cc6-44cd-a5af-ff21f896a42f" containerName="splitter-graph-865f9"
Jul 09 18:55:58.690340 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.690318    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.693167 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.693146    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8wgtz\"/\"default-dockercfg-k5z4r\""
Jul 09 18:55:58.693263 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.693146    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8wgtz\"/\"kube-root-ca.crt\""
Jul 09 18:55:58.694271 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.694251    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8wgtz\"/\"openshift-service-ca.crt\""
Jul 09 18:55:58.702465 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.702445    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"]
Jul 09 18:55:58.752686 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.752651    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-sys\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.752827 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.752761    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-podres\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.752827 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.752789    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rw9b\" (UniqueName: \"kubernetes.io/projected/02909fc4-e6f2-46b0-9dff-81607a60e5fe-kube-api-access-6rw9b\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.752921 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.752826    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-lib-modules\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.752921 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.752846    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-proc\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.853483 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.853450    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-sys\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.853628 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.853527    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-podres\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.853628 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.853548    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rw9b\" (UniqueName: \"kubernetes.io/projected/02909fc4-e6f2-46b0-9dff-81607a60e5fe-kube-api-access-6rw9b\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.853628 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.853566    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-lib-modules\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.853628 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.853575    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-sys\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.853628 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.853589    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-proc\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.853846 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.853644    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-proc\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.853846 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.853707    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-lib-modules\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.853846 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.853735    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02909fc4-e6f2-46b0-9dff-81607a60e5fe-podres\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.863356 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.863333    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rw9b\" (UniqueName: \"kubernetes.io/projected/02909fc4-e6f2-46b0-9dff-81607a60e5fe-kube-api-access-6rw9b\") pod \"perf-node-gather-daemonset-2btw4\" (UID: \"02909fc4-e6f2-46b0-9dff-81607a60e5fe\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:58.999494 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:58.999422    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:55:59.120330 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:59.120298    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"]
Jul 09 18:55:59.123417 ip-10-0-140-234 kubenswrapper[2568]: W0709 18:55:59.123388    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod02909fc4_e6f2_46b0_9dff_81607a60e5fe.slice/crio-6c72c5f08d1b3cf8a651aed5e600fceca43ffbec9ebf3e83741c2f177572931d WatchSource:0}: Error finding container 6c72c5f08d1b3cf8a651aed5e600fceca43ffbec9ebf3e83741c2f177572931d: Status 404 returned error can't find the container with id 6c72c5f08d1b3cf8a651aed5e600fceca43ffbec9ebf3e83741c2f177572931d
Jul 09 18:55:59.476662 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:59.476634    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-46xvq_2b91a8a8-43dc-4f19-b55c-62e2b06801f2/dns/0.log"
Jul 09 18:55:59.496766 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:59.496743    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-46xvq_2b91a8a8-43dc-4f19-b55c-62e2b06801f2/kube-rbac-proxy/0.log"
Jul 09 18:55:59.608928 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:55:59.608882    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kg2kp_24b02b7c-f16a-4f2b-b2cc-2e2b22d22a10/dns-node-resolver/0.log"
Jul 09 18:56:00.073368 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:00.073329    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4" event={"ID":"02909fc4-e6f2-46b0-9dff-81607a60e5fe","Type":"ContainerStarted","Data":"44cb3793a6209a96055a30addb5b6899ecdada33c20d16a73c0b09a1ee30be8e"}
Jul 09 18:56:00.073577 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:00.073384    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4" event={"ID":"02909fc4-e6f2-46b0-9dff-81607a60e5fe","Type":"ContainerStarted","Data":"6c72c5f08d1b3cf8a651aed5e600fceca43ffbec9ebf3e83741c2f177572931d"}
Jul 09 18:56:00.073577 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:00.073464    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:56:00.090825 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:00.090776    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4" podStartSLOduration=2.090761875 podStartE2EDuration="2.090761875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:56:00.088746606 +0000 UTC m=+3114.422610685" watchObservedRunningTime="2026-07-09 18:56:00.090761875 +0000 UTC m=+3114.424625955"
Jul 09 18:56:00.205677 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:00.205649    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jlzk7_b3bd61c5-6cd4-434e-8b3d-487cbacf897f/node-ca/0.log"
Jul 09 18:56:01.423187 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:01.423161    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gjkgm_1cae8e1a-1640-4639-8783-74995450b5a3/serve-healthcheck-canary/0.log"
Jul 09 18:56:01.979311 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:01.979283    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wjp96_1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3/kube-rbac-proxy/0.log"
Jul 09 18:56:01.999341 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:01.999322    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wjp96_1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3/exporter/0.log"
Jul 09 18:56:02.024183 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:02.024160    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wjp96_1cd8fc5a-20f1-4ef9-a4c2-6ac79bc92cd3/extractor/0.log"
Jul 09 18:56:04.040391 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:04.040360    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-748bdf6488-jq2q7_5b3ebd34-180d-40e4-943f-47105daf1629/manager/0.log"
Jul 09 18:56:04.069789 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:04.069760    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-j4v5l_49a22f36-fe11-4c4c-aa8d-19af1a508da2/manager/0.log"
Jul 09 18:56:04.092642 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:04.092618    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-9kvtv_469358e6-fcef-42e1-9f26-5d97e1eba39f/server/0.log"
Jul 09 18:56:06.085545 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:06.085519    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-2btw4"
Jul 09 18:56:09.847485 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:09.847463    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-npq7x_696c7d0f-9dad-4b5e-a4ab-081b032a3bf7/kube-multus-additional-cni-plugins/0.log"
Jul 09 18:56:09.883708 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:09.883681    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-npq7x_696c7d0f-9dad-4b5e-a4ab-081b032a3bf7/egress-router-binary-copy/0.log"
Jul 09 18:56:09.911983 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:09.911962    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-npq7x_696c7d0f-9dad-4b5e-a4ab-081b032a3bf7/cni-plugins/0.log"
Jul 09 18:56:09.944347 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:09.944324    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-npq7x_696c7d0f-9dad-4b5e-a4ab-081b032a3bf7/bond-cni-plugin/0.log"
Jul 09 18:56:09.975297 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:09.975276    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-npq7x_696c7d0f-9dad-4b5e-a4ab-081b032a3bf7/routeoverride-cni/0.log"
Jul 09 18:56:10.006667 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:10.006614    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-npq7x_696c7d0f-9dad-4b5e-a4ab-081b032a3bf7/whereabouts-cni-bincopy/0.log"
Jul 09 18:56:10.040659 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:10.040636    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-npq7x_696c7d0f-9dad-4b5e-a4ab-081b032a3bf7/whereabouts-cni/0.log"
Jul 09 18:56:10.331673 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:10.331649    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wvqdj_148bb254-e170-4306-9b3f-10811338d4b2/kube-multus/0.log"
Jul 09 18:56:10.404555 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:10.404527    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f84z2_c4b61885-5a4f-4355-ab6c-f95538b7992b/network-metrics-daemon/0.log"
Jul 09 18:56:10.440694 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:10.440668    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f84z2_c4b61885-5a4f-4355-ab6c-f95538b7992b/kube-rbac-proxy/0.log"
Jul 09 18:56:11.536028 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:11.535993    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-controller/0.log"
Jul 09 18:56:11.551948 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:11.551927    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/1.log"
Jul 09 18:56:11.552093 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:11.551967    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovn-acl-logging/0.log"
Jul 09 18:56:11.570742 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:11.570722    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/kube-rbac-proxy-node/0.log"
Jul 09 18:56:11.591283 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:11.591255    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 18:56:11.606811 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:11.606792    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/northd/0.log"
Jul 09 18:56:11.627772 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:11.627752    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/nbdb/0.log"
Jul 09 18:56:11.648562 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:11.648540    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/sbdb/0.log"
Jul 09 18:56:11.687815 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:11.687787    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bstvh_73fdebde-0b8f-486b-aa5c-ef4b20aafffd/ovnkube-controller/0.log"
Jul 09 18:56:12.947128 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:12.947098    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bth88_113c3ee7-a95e-42b6-87d8-68149f3382c8/network-check-target-container/0.log"
Jul 09 18:56:13.983934 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:13.983902    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-bkn7f_ea4d5a34-f150-4a58-ad65-60b4c5f57295/iptables-alerter/0.log"
Jul 09 18:56:14.667977 ip-10-0-140-234 kubenswrapper[2568]: I0709 18:56:14.667948    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-wvzlz_cda30b50-0c11-4e28-8ea9-a954ae1b8089/tuned/0.log"