Jul 01 19:19:10.585085 ip-10-0-141-93 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 19:19:10.585094 ip-10-0-141-93 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 19:19:10.585101 ip-10-0-141-93 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 19:19:10.585333 ip-10-0-141-93 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 19:19:20.731268 ip-10-0-141-93 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 19:19:20.731285 ip-10-0-141-93 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4d9209a3ed6c429fa5347d164a8b1976 --
Jul 01 19:21:25.341129 ip-10-0-141-93 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.713868    2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719136    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719155    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719159    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719162    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719165    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719169    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719172    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:25.791219 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719174    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:25.783911 ip-10-0-141-93 systemd[1]: Started Kubernetes Kubelet.
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719178    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719188    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719191    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719194    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719197    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719200    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719203    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719205    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719210    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719213    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719216    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719218    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719221    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719224    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719230    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719233    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719236    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719241    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:25.792174 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719245    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719248    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719251    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719254    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719257    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719259    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719262    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719265    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719270    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719273    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719275    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719278    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719281    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719284    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719286    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719290    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719294    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719297    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719300    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719303    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:25.793484 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719305    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719311    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719313    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719318    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719322    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719324    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719327    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719331    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719334    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719337    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719339    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719342    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719345    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719351    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719354    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719357    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719361    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719364    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719367    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719369    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:25.794168 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719372    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719375    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719377    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719380    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719383    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719388    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719390    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719394    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719398    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719403    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719409    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719414    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719417    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719420    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719423    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719427    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719433    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719436    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719439    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719441    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:25.794777 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719444    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719446    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719449    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719452    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719455    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719458    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719461    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719464    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719466    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719472    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719475    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719478    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.719480    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720796    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720808    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720811    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720814    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720817    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720820    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720823    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:25.795401 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720826    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720829    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720832    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720835    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720838    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720841    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720843    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720846    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720848    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720852    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720855    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720858    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720860    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720863    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720866    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720868    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720872    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720874    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720877    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720880    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:25.796201 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720883    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720886    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720889    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720893    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720897    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720900    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720903    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720906    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720908    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720911    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720914    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720917    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720919    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720922    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720924    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720927    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720929    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720932    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720935    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:25.796824 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720938    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720941    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720943    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720946    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720950    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720952    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720957    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720961    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720964    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720967    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720970    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720973    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720977    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720980    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720983    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720986    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720989    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720992    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720995    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:26.004336 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.720998    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721001    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721004    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721007    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721010    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721014    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721018    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721021    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721024    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721026    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721029    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721032    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721035    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721039    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721042    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721045    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721048    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721051    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721054    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:26.022850 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721057    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721060    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721063    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721065    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721068    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721070    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721073    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721075    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721078    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721083    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721086    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721089    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721091    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721094    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721096    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721482    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721488    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721491    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721493    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721496    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:26.024200 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721499    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721501    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721504    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721507    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721509    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721514    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721517    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721520    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721523    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721525    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721528    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721531    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721533    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721536    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721539    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721542    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721544    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721546    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721549    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721553    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:26.025320 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721557    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721562    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721565    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721568    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721570    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721573    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721575    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721578    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721581    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721583    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721586    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721588    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721591    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721593    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721596    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721598    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721601    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721603    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721606    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721609    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:26.026462 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721611    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721614    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721616    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721618    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721622    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721624    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721626    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721629    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721632    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721634    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721637    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721643    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721646    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721650    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721653    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721656    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721658    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721661    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721664    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:26.027610 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721666    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721669    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721671    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721674    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721677    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721699    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721703    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721706    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721708    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721711    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721714    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721716    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721719    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721721    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721724    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721727    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721729    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721732    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721736    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721738    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:26.028552 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721741    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721744    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721746    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721748    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721752    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721755    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721759    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721762    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721765    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721767    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721770    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721772    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721775    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721777    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.721780    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721848    2586 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721855    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721869    2586 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721873    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721878    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721881    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 19:21:26.029700 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721886    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721890    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721893    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721896    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721900    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721903    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721906    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721909    2586 flags.go:64] FLAG: --cgroup-root=""
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721912    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721915    2586 flags.go:64] FLAG: --client-ca-file=""
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721919    2586 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721922    2586 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721925    2586 flags.go:64] FLAG: --cluster-domain=""
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721928    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721931    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721936    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721939    2586 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721944    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721947    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721950    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721953    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721956    2586 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721959    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721962    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 19:21:26.030849 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721965    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721968    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721972    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721975    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721978    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721981    2586 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721984    2586 flags.go:64] FLAG: --enable-server="true"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721987    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721991    2586 flags.go:64] FLAG: --event-burst="100"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721995    2586 flags.go:64] FLAG: --event-qps="50"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.721998    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722001    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722003    2586 flags.go:64] FLAG: --eviction-hard=""
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722007    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722010    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722013    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722016    2586 flags.go:64] FLAG: --eviction-soft=""
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722019    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722022    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722025    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722028    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722031    2586 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722034    2586 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722038    2586 flags.go:64] FLAG: --feature-gates=""
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722042    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 19:21:26.031932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722046    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722050    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722053    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722056    2586 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722059    2586 flags.go:64] FLAG: --help="false"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722062    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722065    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722068    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722071    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722074    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722078    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722081    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722084    2586 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722086    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722094    2586 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722097    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722100    2586 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722103    2586 flags.go:64] FLAG: --kube-reserved=""
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722106    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722108    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722111    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722114    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722117    2586 flags.go:64] FLAG: --lock-file=""
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722120    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 19:21:26.033299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722123    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722126    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722131    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722135    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722138    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722141    2586 flags.go:64] FLAG: --logging-format="text"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722145    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722148    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722153    2586 flags.go:64] FLAG: --manifest-url=""
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722156    2586 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722160    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722163    2586 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722167    2586 flags.go:64] FLAG: --max-pods="110"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722171    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722174    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722177    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722180    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722183    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722185    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722188    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722195    2586 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722198    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722201    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 19:21:26.034479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722204    2586 flags.go:64] FLAG: --pod-cidr=""
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722207    2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722214    2586 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722217    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722220    2586 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722223    2586 flags.go:64] FLAG: --port="10250"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722225    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722229    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0871e10db5e78c00a"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722232    2586 flags.go:64] FLAG: --qos-reserved=""
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722235    2586 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722238    2586 flags.go:64] FLAG: --register-node="true"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722241    2586 flags.go:64] FLAG: --register-with-taints=""
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722244    2586 flags.go:64] FLAG: --registry-burst="10"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722248    2586 flags.go:64] FLAG: --registry-qps="5"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722251    2586 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722253    2586 flags.go:64] FLAG: --reserved-memory=""
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722258    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722262    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722265    2586 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722268    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722271    2586 flags.go:64] FLAG: --runonce="false"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722274    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722277    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722279    2586 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722283    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 19:21:26.035626 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722285    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722288    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722291    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722294    2586 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722297    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722300    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722303    2586 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722306    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722309    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722312    2586 flags.go:64] FLAG: --system-cgroups=""
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722314    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722320    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722322    2586 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722325    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722329    2586 flags.go:64] FLAG: --tls-min-version=""
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722331    2586 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722334    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722337    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722340    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722343    2586 flags.go:64] FLAG: --v="2"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722347    2586 flags.go:64] FLAG: --version="false"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722351    2586 flags.go:64] FLAG: --vmodule=""
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722355    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722359    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722477    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:26.036795 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722481    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722484    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722487    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722490    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722493    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722496    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722498    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722501    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722504    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722506    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722509    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722512    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722514    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722517    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722519    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722522    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722525    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722528    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722530    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722533    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:26.040206 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722535    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722538    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722541    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722543    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722546    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722548    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722551    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722553    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722556    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722559    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722563    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722567    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722570    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722573    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722575    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722578    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722581    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722583    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722586    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722588    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:26.041358 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722591    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722593    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722596    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722598    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722601    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722603    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722606    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722608    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722612    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722616    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722618    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722621    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722624    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722626    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722629    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722631    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722634    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722636    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722639    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:26.046667 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722641    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722644    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722646    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722651    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722656    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722660    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722663    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722666    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722668    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722671    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722674    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722676    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722679    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722700    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722704    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722707    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722709    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722712    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722714    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:26.047326 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722717    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722720    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722722    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722725    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722729    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722732    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722735    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722738    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722741    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722744    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722747    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722749    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722752    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722755    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722757    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722760    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722764    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722769    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722772    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:26.047910 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.722774    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.722780    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.729483    2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.729498    2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.729512    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.729522    2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729578    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729583    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729586    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729590    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729593    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729596    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729600    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729603    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:26.048507 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729606    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729609    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729612    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729616    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729619    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729622    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729625    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729627    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729630    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729633    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729635    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729640    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729644    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729647    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729650    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729653    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729656    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729659    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729662    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:26.048962 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729665    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729668    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729671    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729674    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729676    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729679    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729696    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729699    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729702    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729705    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729708    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729713    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729717    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729720    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729723    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729725    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729728    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729731    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729734    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729737    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:26.261186 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729740    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729743    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729746    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729750    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729754    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729757    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729760    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729763    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729766    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729772    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729775    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729778    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729781    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729784    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729787    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729793    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729796    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729799    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729802    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:26.262409 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729806    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729809    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729812    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729815    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729818    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729820    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729823    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729825    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729828    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729830    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729833    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729835    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729838    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729840    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729843    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729845    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729848    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729851    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729853    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729856    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:26.263533 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729858    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729861    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729865    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729868    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729870    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729873    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729876    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729878    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729881    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729884    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729886    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729889    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.729893    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.729898    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730018    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:26.264690 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730024    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730027    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730030    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730033    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730036    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730039    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730042    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730045    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730047    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730050    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730053    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730055    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730058    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730061    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730063    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730067    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730071    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730074    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730077    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730080    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:26.265593 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730084    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730087    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730090    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730093    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730096    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730099    2586 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730102    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730106    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730109    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730113    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730116    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730119    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730121    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730124    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730127    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730129    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730133    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730136    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730140    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:26.266515 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730143    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730146    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730149    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730151    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730154    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730157    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730160    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730163    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730165    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730168    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730170    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730173    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730176    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730180    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730183    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730185    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730188    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730190    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730193    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730196    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:26.267461 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730199    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730202    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730207    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730211    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730213    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730216    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730219    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730221    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730224    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730226    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730229    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730231    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730234    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730237    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730239    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730242    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730245    2586 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730247    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730249    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:26.268385 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730252    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730254    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730257    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730260    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730262    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730265    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730269    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730271    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730274    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730276    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730279    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730282    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730284    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730287    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730289    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730294    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730296    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730299    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730302    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:26.269294 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:25.730305    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.730309    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.730423    2586 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.732667    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.734024    2586 server.go:1019] "Starting client certificate rotation"
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.734131    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.734837    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.736573    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.739038    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.749891    2586 log.go:25] "Validated CRI v1 runtime API"
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.752737    2586 log.go:25] "Validated CRI v1 image API"
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.753455    2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 19:21:26.270215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.755708    2586 fs.go:135] Filesystem UUIDs: map[22fb8b45-8618-4b0d-9c86-37138572bd3d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8563bcf3-fded-4ca8-a2ff-8796559c6ec9:/dev/nvme0n1p4]
Jul 01 19:21:26.270892 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.755724    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 19:21:26.271139 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.761018    2586 manager.go:216] Machine: {Timestamp:2026-07-01 19:21:25.759825078 +0000 UTC m=+0.306074403 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097343 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ed0a9e1fb384eb7f1575c0641ef31 SystemUUID:ec2ed0a9-e1fb-384e-b7f1-575c0641ef31 BootID:4d9209a3-ed6c-429f-a534-7d164a8b1976 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:61:53:87:a9:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:61:53:87:a9:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:87:dd:14:30:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 19:21:26.271139 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.761130    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 19:21:26.271139 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.761206    2586 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 19:21:26.271139 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.763430    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 19:21:26.271139 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.764169    2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 19:21:26.271139 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.764195    2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-93.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 19:21:26.271139 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.764992    2586 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.765002    2586 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.765010    2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.765030    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.766174    2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.769742    2586 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.769859    2586 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.769887    2586 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.769901    2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.769921    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.769932    2586 kubelet.go:398] "Adding apiserver pod source"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.769947    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.770303    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.770319    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.772145    2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.773931    2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.773966    2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774882    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774898    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774907    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774915    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774927    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774941    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774953    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774962    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774971    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774979    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774990    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.774998    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.775798    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.775818    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.779893    2586 server.go:1268] "Started kubelet"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.779973    2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.780228    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.783423    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.783792    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-93.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.783800    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.784020    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.785473    2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.786994    2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 19:21:26.271594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.787665    2586 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.794741    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-93.ec2.internal.18be40bda6fc9aa0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-93.ec2.internal,UID:ip-10-0-141-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-93.ec2.internal,},FirstTimestamp:2026-07-01 19:21:25.779864224 +0000 UTC m=+0.326113552,LastTimestamp:2026-07-01 19:21:25.779864224 +0000 UTC m=+0.326113552,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-93.ec2.internal,}"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.798835    2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.798856    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.799052    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-93.ec2.internal\" not found"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.799622    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.799668    2586 factory.go:55] Registering systemd factory
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.799674    2586 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.799710    2586 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.799699    2586 factory.go:223] Registration of the systemd container factory successfully
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.801000    2586 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.801011    2586 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.801320    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7784l"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.801422    2586 factory.go:153] Registering CRI-O factory
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.801440    2586 factory.go:223] Registration of the crio container factory successfully
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.801506    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.801528    2586 factory.go:103] Registering Raw factory
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.801541    2586 manager.go:1196] Started watching for new ooms in manager
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.802107    2586 manager.go:349] Starting recovery of all containers
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.802567    2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.802664    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.810271    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7784l"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.816920    2586 manager.go:354] Recovery completed
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.826670    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.828487    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.828513    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.828524    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.829084    2586 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.829091    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.829096    2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.829123    2586 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.829828    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:26.273429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.829855    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.829870    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.830305    2586 policy_none.go:49] "None policy: Start"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.830319    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.830330    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.831210    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-93.ec2.internal.18be40bda9e2b420  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-93.ec2.internal,UID:ip-10-0-141-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-93.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-93.ec2.internal,},FirstTimestamp:2026-07-01 19:21:25.828498464 +0000 UTC m=+0.374747793,LastTimestamp:2026-07-01 19:21:25.828498464 +0000 UTC m=+0.374747793,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-93.ec2.internal,}"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.831581    2586 policy_none.go:47] "Start"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.866439    2586 manager.go:337] "Starting Device Plugin manager"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.866478    2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.866489    2586 server.go:87] "Starting device plugin registration server"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.866814    2586 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.866829    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.866946    2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.867020    2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.867030    2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.867513    2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.867574    2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-93.ec2.internal\" not found"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.876182    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.877691    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.877728    2586 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.877748    2586 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.877795    2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.880436    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.972224    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.973315    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.973341    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.973352    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.973376    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.978541    2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-93.ec2.internal"]
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.978610    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.979983    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.980009    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.980019    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:26.274518 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.980580    2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:25.980598    2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-93.ec2.internal\": node \"ip-10-0-141-93.ec2.internal\" not found"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.981384    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.981499    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.981531    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.982114    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.982140    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.982151    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.982114    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.982223    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.982235    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.983357    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.983378    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.984071    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.984098    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:25.984109    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.001152    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-93.ec2.internal\" not found"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.002291    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f59f55115b683b9b149f1e104e483009-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal\" (UID: \"f59f55115b683b9b149f1e104e483009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.002312    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ea467cf493fb8782e204574fc06f1c8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-93.ec2.internal\" (UID: \"4ea467cf493fb8782e204574fc06f1c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.002329    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f59f55115b683b9b149f1e104e483009-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal\" (UID: \"f59f55115b683b9b149f1e104e483009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.017732    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-93.ec2.internal\" not found" node="ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.022435    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-93.ec2.internal\" not found" node="ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.101429    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-93.ec2.internal\" not found"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.102624    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f59f55115b683b9b149f1e104e483009-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal\" (UID: \"f59f55115b683b9b149f1e104e483009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.278227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.102649    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f59f55115b683b9b149f1e104e483009-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal\" (UID: \"f59f55115b683b9b149f1e104e483009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.279386 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.102667    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ea467cf493fb8782e204574fc06f1c8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-93.ec2.internal\" (UID: \"4ea467cf493fb8782e204574fc06f1c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.279386 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.102722    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ea467cf493fb8782e204574fc06f1c8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-93.ec2.internal\" (UID: \"4ea467cf493fb8782e204574fc06f1c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.279386 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.102766    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f59f55115b683b9b149f1e104e483009-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal\" (UID: \"f59f55115b683b9b149f1e104e483009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.279386 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.102790    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f59f55115b683b9b149f1e104e483009-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal\" (UID: \"f59f55115b683b9b149f1e104e483009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.279386 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.202461    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-93.ec2.internal\" not found"
Jul 01 19:21:26.537517 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.303127    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-93.ec2.internal\" not found"
Jul 01 19:21:26.537517 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.322793    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.537517 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.327293    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.537517 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.404097    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-93.ec2.internal\" not found"
Jul 01 19:21:26.537517 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.505201    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-93.ec2.internal\" not found"
Jul 01 19:21:26.782500 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.605402    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-93.ec2.internal\" not found"
Jul 01 19:21:26.782500 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.705579    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-93.ec2.internal\" not found"
Jul 01 19:21:26.782500 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.733917    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 19:21:26.782500 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.734095    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 19:21:26.782500 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.745390    2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:26.782500 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.771191    2586 apiserver.go:52] "Watching apiserver"
Jul 01 19:21:26.788466 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.786309    2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 19:21:26.788466 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.788239    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-56r4t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w","openshift-cluster-node-tuning-operator/tuned-gklml","openshift-dns/node-resolver-gd8x7","openshift-image-registry/node-ca-nrf8m","openshift-multus/network-metrics-daemon-jbvd8","openshift-network-operator/iptables-alerter-ktjxz","kube-system/global-pull-secret-syncer-5zvjf","openshift-multus/multus-52gwt","openshift-multus/multus-additional-cni-plugins-b29mt","openshift-network-diagnostics/network-check-target-bc752","openshift-ovn-kubernetes/ovnkube-node-bxjj5"]
Jul 01 19:21:26.790478 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.789876    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:26.791277 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.791156    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.792678    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.794164    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gd8x7"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.795187    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrf8m"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.796317    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.796369    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.797526    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ktjxz"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.798681    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.798767    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.799159    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.799190    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.800104    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.801587    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.803016    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.803100    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.804488    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808056    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-var-lib-kubelet\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808095    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c775b400-2625-4656-a442-a8069e8bd20e-etc-tuned\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808120    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qmr\" (UniqueName: \"kubernetes.io/projected/0723c42e-0fc4-4c03-b5e6-83d0af6a9488-kube-api-access-88qmr\") pod \"node-resolver-gd8x7\" (UID: \"0723c42e-0fc4-4c03-b5e6-83d0af6a9488\") " pod="openshift-dns/node-resolver-gd8x7"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808159    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-run-k8s-cni-cncf-io\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.950970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808202    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-run-netns\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808235    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-var-lib-cni-multus\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808260    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2bc24c26-f688-41bb-ad3e-27d184fe1669-iptables-alerter-script\") pod \"iptables-alerter-ktjxz\" (UID: \"2bc24c26-f688-41bb-ad3e-27d184fe1669\") " pod="openshift-network-operator/iptables-alerter-ktjxz"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808287    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-sysconfig\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808312    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808335    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55e6ffec-a820-4802-89cb-747770b45bb9-dbus\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808365    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808391    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxkh\" (UniqueName: \"kubernetes.io/projected/a0e28d47-acbf-4c6b-ad83-a883ad149185-kube-api-access-htxkh\") pod \"node-ca-nrf8m\" (UID: \"a0e28d47-acbf-4c6b-ad83-a883ad149185\") " pod="openshift-image-registry/node-ca-nrf8m"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808418    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808443    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-cni-binary-copy\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808471    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-hostroot\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808492    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-sysctl-conf\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808506    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808518    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808444    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808510    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0723c42e-0fc4-4c03-b5e6-83d0af6a9488-hosts-file\") pod \"node-resolver-gd8x7\" (UID: \"0723c42e-0fc4-4c03-b5e6-83d0af6a9488\") " pod="openshift-dns/node-resolver-gd8x7"
Jul 01 19:21:26.952751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808578    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-var-lib-cni-bin\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808607    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808606    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-sysctl-d\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808645    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808764    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-var-lib-kubelet\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808788    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-run-multus-certs\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808806    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-lib-modules\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808823    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bc24c26-f688-41bb-ad3e-27d184fe1669-host-slash\") pod \"iptables-alerter-ktjxz\" (UID: \"2bc24c26-f688-41bb-ad3e-27d184fe1669\") " pod="openshift-network-operator/iptables-alerter-ktjxz"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808840    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-daemon-config\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808862    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25fkp\" (UniqueName: \"kubernetes.io/projected/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-kube-api-access-25fkp\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808878    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9dd6796f-9e3d-4c78-93ec-acd06d798d3c-konnectivity-ca\") pod \"konnectivity-agent-56r4t\" (UID: \"9dd6796f-9e3d-4c78-93ec-acd06d798d3c\") " pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808893    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0e28d47-acbf-4c6b-ad83-a883ad149185-serviceca\") pod \"node-ca-nrf8m\" (UID: \"a0e28d47-acbf-4c6b-ad83-a883ad149185\") " pod="openshift-image-registry/node-ca-nrf8m"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808923    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-etc-kubernetes\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808953    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhd4\" (UniqueName: \"kubernetes.io/projected/2bc24c26-f688-41bb-ad3e-27d184fe1669-kube-api-access-wwhd4\") pod \"iptables-alerter-ktjxz\" (UID: \"2bc24c26-f688-41bb-ad3e-27d184fe1669\") " pod="openshift-network-operator/iptables-alerter-ktjxz"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808976    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.953692 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.808999    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-modprobe-d\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809020    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-run\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809045    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c775b400-2625-4656-a442-a8069e8bd20e-tmp\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809066    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809090    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0723c42e-0fc4-4c03-b5e6-83d0af6a9488-tmp-dir\") pod \"node-resolver-gd8x7\" (UID: \"0723c42e-0fc4-4c03-b5e6-83d0af6a9488\") " pod="openshift-dns/node-resolver-gd8x7"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809124    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-cni-dir\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809156    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-conf-dir\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809180    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq958\" (UniqueName: \"kubernetes.io/projected/c775b400-2625-4656-a442-a8069e8bd20e-kube-api-access-rq958\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809227    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-systemd\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809250    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55e6ffec-a820-4802-89cb-747770b45bb9-kubelet-config\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809272    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-cnibin\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809366    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-socket-dir-parent\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809389    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-sys\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809417    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0e28d47-acbf-4c6b-ad83-a883ad149185-host\") pod \"node-ca-nrf8m\" (UID: \"a0e28d47-acbf-4c6b-ad83-a883ad149185\") " pod="openshift-image-registry/node-ca-nrf8m"
Jul 01 19:21:26.954832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809440    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-socket-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809456    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-sys-fs\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809471    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-device-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809488    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-kubernetes\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809503    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-host\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809521    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn8h6\" (UniqueName: \"kubernetes.io/projected/f41077b2-239b-44f8-be7d-b8b9464b42e9-kube-api-access-bn8h6\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809548    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-system-cni-dir\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809563    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-os-release\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809591    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9dd6796f-9e3d-4c78-93ec-acd06d798d3c-agent-certs\") pod \"konnectivity-agent-56r4t\" (UID: \"9dd6796f-9e3d-4c78-93ec-acd06d798d3c\") " pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809613    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-registration-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809649    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5znf6\" (UniqueName: \"kubernetes.io/projected/4173f069-b400-4be2-a30f-99c0d6b077cf-kube-api-access-5znf6\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809852    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809872    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809909    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9wm8s\""
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.809936    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.810237    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.810312    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.810337    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 19:21:26.955716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.810420    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.810557    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.810751    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.810819    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.810891    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.810939    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.810958    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.811018    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.811021    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.811623    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rtv6f\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.811691    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.811693    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6ksd6\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.811758    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.811912    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.811967    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-frpb9\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.812143    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n52r5\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.812351    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9szs4\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.812442    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.812575    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n8x4c\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.812716    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.812760    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.812812    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5cbjg\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.813167    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-smxx9\""
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.814867    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 19:16:26 +0000 UTC" deadline="2028-02-11 11:52:59.948926749 +0000 UTC"
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.814917    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14152h31m33.134014265s"
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.823745    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.826264    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.826365    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.ec2.internal"
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.826613    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal"]
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.839809    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-93.ec2.internal"]
Jul 01 19:21:26.956876 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.840449    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.863837    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2sxmp"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.875594    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2sxmp"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.900556    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.909974    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-cni-dir\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.909998    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-conf-dir\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910016    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq958\" (UniqueName: \"kubernetes.io/projected/c775b400-2625-4656-a442-a8069e8bd20e-kube-api-access-rq958\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910047    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d69d7279-1771-42d1-bde5-3dbc8ee7f990-ovnkube-config\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910074    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-os-release\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910119    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-systemd\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910115    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-conf-dir\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910148    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-cni-dir\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910161    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55e6ffec-a820-4802-89cb-747770b45bb9-kubelet-config\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910200    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-run-ovn\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910195    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-systemd\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910223    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55e6ffec-a820-4802-89cb-747770b45bb9-kubelet-config\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:26.958456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910241    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-cnibin\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910272    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-socket-dir-parent\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910286    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-cnibin\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910297    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-sys\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910324    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0e28d47-acbf-4c6b-ad83-a883ad149185-host\") pod \"node-ca-nrf8m\" (UID: \"a0e28d47-acbf-4c6b-ad83-a883ad149185\") " pod="openshift-image-registry/node-ca-nrf8m"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910345    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-etc-openvswitch\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910362    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-run-ovn-kubernetes\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910366    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-socket-dir-parent\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910382    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5a828de-7dd6-45af-b69c-1837becfe480-cni-binary-copy\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910362    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-sys\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910399    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0e28d47-acbf-4c6b-ad83-a883ad149185-host\") pod \"node-ca-nrf8m\" (UID: \"a0e28d47-acbf-4c6b-ad83-a883ad149185\") " pod="openshift-image-registry/node-ca-nrf8m"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910417    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-socket-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910434    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-sys-fs\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910452    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d69d7279-1771-42d1-bde5-3dbc8ee7f990-ovnkube-script-lib\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.959310 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910480    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-device-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910506    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-sys-fs\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910511    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-kubernetes\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910546    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-host\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910562    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-socket-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910575    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn8h6\" (UniqueName: \"kubernetes.io/projected/f41077b2-239b-44f8-be7d-b8b9464b42e9-kube-api-access-bn8h6\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910593    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-host\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910600    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-kubernetes\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910605    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-system-cni-dir\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910597    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-device-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910648    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-system-cni-dir\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910730    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-os-release\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910752    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-system-cni-dir\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910796    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9dd6796f-9e3d-4c78-93ec-acd06d798d3c-agent-certs\") pod \"konnectivity-agent-56r4t\" (UID: \"9dd6796f-9e3d-4c78-93ec-acd06d798d3c\") " pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910836    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-os-release\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910833    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-registration-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910940    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-registration-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910953    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5znf6\" (UniqueName: \"kubernetes.io/projected/4173f069-b400-4be2-a30f-99c0d6b077cf-kube-api-access-5znf6\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.910981    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-systemd-units\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911077    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-cni-netd\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911102    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-var-lib-kubelet\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c775b400-2625-4656-a442-a8069e8bd20e-etc-tuned\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911150    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-var-lib-openvswitch\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911168    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wr7g\" (UniqueName: \"kubernetes.io/projected/d69d7279-1771-42d1-bde5-3dbc8ee7f990-kube-api-access-5wr7g\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911192    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qmr\" (UniqueName: \"kubernetes.io/projected/0723c42e-0fc4-4c03-b5e6-83d0af6a9488-kube-api-access-88qmr\") pod \"node-resolver-gd8x7\" (UID: \"0723c42e-0fc4-4c03-b5e6-83d0af6a9488\") " pod="openshift-dns/node-resolver-gd8x7"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911257    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-run-k8s-cni-cncf-io\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911269    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-var-lib-kubelet\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911290    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-run-netns\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911313    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-var-lib-cni-multus\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.961119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911336    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-run-k8s-cni-cncf-io\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911343    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2bc24c26-f688-41bb-ad3e-27d184fe1669-iptables-alerter-script\") pod \"iptables-alerter-ktjxz\" (UID: \"2bc24c26-f688-41bb-ad3e-27d184fe1669\") " pod="openshift-network-operator/iptables-alerter-ktjxz"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911377    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-run-netns\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911368    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-var-lib-cni-multus\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911396    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-run-systemd\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911433    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911461    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d69d7279-1771-42d1-bde5-3dbc8ee7f990-env-overrides\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911475    2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911497    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-sysconfig\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911524    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911548    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55e6ffec-a820-4802-89cb-747770b45bb9-dbus\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911560    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-sysconfig\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911572    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911598    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxkh\" (UniqueName: \"kubernetes.io/projected/a0e28d47-acbf-4c6b-ad83-a883ad149185-kube-api-access-htxkh\") pod \"node-ca-nrf8m\" (UID: \"a0e28d47-acbf-4c6b-ad83-a883ad149185\") " pod="openshift-image-registry/node-ca-nrf8m"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911624    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-log-socket\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.962030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911645    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55e6ffec-a820-4802-89cb-747770b45bb9-dbus\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911650    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-cni-binary-copy\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911678    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-hostroot\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.911678    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911735    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-hostroot\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911737    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-sysctl-conf\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911781    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0723c42e-0fc4-4c03-b5e6-83d0af6a9488-hosts-file\") pod \"node-resolver-gd8x7\" (UID: \"0723c42e-0fc4-4c03-b5e6-83d0af6a9488\") " pod="openshift-dns/node-resolver-gd8x7"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911810    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5a828de-7dd6-45af-b69c-1837becfe480-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911865    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0723c42e-0fc4-4c03-b5e6-83d0af6a9488-hosts-file\") pod \"node-resolver-gd8x7\" (UID: \"0723c42e-0fc4-4c03-b5e6-83d0af6a9488\") " pod="openshift-dns/node-resolver-gd8x7"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.911881    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs podName:f41077b2-239b-44f8-be7d-b8b9464b42e9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:27.411836975 +0000 UTC m=+1.958086303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs") pod "network-metrics-daemon-jbvd8" (UID: "f41077b2-239b-44f8-be7d-b8b9464b42e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911893    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-sysctl-conf\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911899    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-var-lib-cni-bin\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911922    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-var-lib-cni-bin\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911929    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-node-log\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.963173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911956    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-sysctl-d\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.911982    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-run-netns\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912028    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5a828de-7dd6-45af-b69c-1837becfe480-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.912055    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912066    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-var-lib-kubelet\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912098    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-run-multus-certs\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912102    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-sysctl-d\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:26.912113    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret podName:55e6ffec-a820-4802-89cb-747770b45bb9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:27.412099774 +0000 UTC m=+1.958349091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret") pod "global-pull-secret-syncer-5zvjf" (UID: "55e6ffec-a820-4802-89cb-747770b45bb9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912051    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2bc24c26-f688-41bb-ad3e-27d184fe1669-iptables-alerter-script\") pod \"iptables-alerter-ktjxz\" (UID: \"2bc24c26-f688-41bb-ad3e-27d184fe1669\") " pod="openshift-network-operator/iptables-alerter-ktjxz"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912151    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-var-lib-kubelet\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912164    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-lib-modules\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912188    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-host-run-multus-certs\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912193    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bc24c26-f688-41bb-ad3e-27d184fe1669-host-slash\") pod \"iptables-alerter-ktjxz\" (UID: \"2bc24c26-f688-41bb-ad3e-27d184fe1669\") " pod="openshift-network-operator/iptables-alerter-ktjxz"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912236    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bc24c26-f688-41bb-ad3e-27d184fe1669-host-slash\") pod \"iptables-alerter-ktjxz\" (UID: \"2bc24c26-f688-41bb-ad3e-27d184fe1669\") " pod="openshift-network-operator/iptables-alerter-ktjxz"
Jul 01 19:21:26.964087 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912239    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-daemon-config\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912275    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25fkp\" (UniqueName: \"kubernetes.io/projected/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-kube-api-access-25fkp\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912303    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9dd6796f-9e3d-4c78-93ec-acd06d798d3c-konnectivity-ca\") pod \"konnectivity-agent-56r4t\" (UID: \"9dd6796f-9e3d-4c78-93ec-acd06d798d3c\") " pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912306    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-lib-modules\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912330    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0e28d47-acbf-4c6b-ad83-a883ad149185-serviceca\") pod \"node-ca-nrf8m\" (UID: \"a0e28d47-acbf-4c6b-ad83-a883ad149185\") " pod="openshift-image-registry/node-ca-nrf8m"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912339    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-cni-binary-copy\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912356    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-slash\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912384    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-cni-bin\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912407    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-cnibin\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912441    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-etc-kubernetes\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912479    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhd4\" (UniqueName: \"kubernetes.io/projected/2bc24c26-f688-41bb-ad3e-27d184fe1669-kube-api-access-wwhd4\") pod \"iptables-alerter-ktjxz\" (UID: \"2bc24c26-f688-41bb-ad3e-27d184fe1669\") " pod="openshift-network-operator/iptables-alerter-ktjxz"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912509    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jz5s\" (UniqueName: \"kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s\") pod \"network-check-target-bc752\" (UID: \"538fff2a-3657-456b-9158-297890173d5c\") " pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912534    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-etc-kubernetes\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912536    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d69d7279-1771-42d1-bde5-3dbc8ee7f990-ovn-node-metrics-cert\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.966164 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912768    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-multus-daemon-config\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.912880    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9dd6796f-9e3d-4c78-93ec-acd06d798d3c-konnectivity-ca\") pod \"konnectivity-agent-56r4t\" (UID: \"9dd6796f-9e3d-4c78-93ec-acd06d798d3c\") " pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.913088    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0e28d47-acbf-4c6b-ad83-a883ad149185-serviceca\") pod \"node-ca-nrf8m\" (UID: \"a0e28d47-acbf-4c6b-ad83-a883ad149185\") " pod="openshift-image-registry/node-ca-nrf8m"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.915633    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9dd6796f-9e3d-4c78-93ec-acd06d798d3c-agent-certs\") pod \"konnectivity-agent-56r4t\" (UID: \"9dd6796f-9e3d-4c78-93ec-acd06d798d3c\") " pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.916404    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c775b400-2625-4656-a442-a8069e8bd20e-etc-tuned\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917170    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfwdb\" (UniqueName: \"kubernetes.io/projected/b5a828de-7dd6-45af-b69c-1837becfe480-kube-api-access-gfwdb\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917201    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917220    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-modprobe-d\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917245    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-run\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917270    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c775b400-2625-4656-a442-a8069e8bd20e-tmp\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917294    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917325    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-etc-modprobe-d\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917353    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917358    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c775b400-2625-4656-a442-a8069e8bd20e-run\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.967073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917395    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4173f069-b400-4be2-a30f-99c0d6b077cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917400    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-kubelet\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917437    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-run-openvswitch\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917473    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917507    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0723c42e-0fc4-4c03-b5e6-83d0af6a9488-tmp-dir\") pod \"node-resolver-gd8x7\" (UID: \"0723c42e-0fc4-4c03-b5e6-83d0af6a9488\") " pod="openshift-dns/node-resolver-gd8x7"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.917841    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0723c42e-0fc4-4c03-b5e6-83d0af6a9488-tmp-dir\") pod \"node-resolver-gd8x7\" (UID: \"0723c42e-0fc4-4c03-b5e6-83d0af6a9488\") " pod="openshift-dns/node-resolver-gd8x7"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.919287    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c775b400-2625-4656-a442-a8069e8bd20e-tmp\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.919871    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn8h6\" (UniqueName: \"kubernetes.io/projected/f41077b2-239b-44f8-be7d-b8b9464b42e9-kube-api-access-bn8h6\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.920066    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq958\" (UniqueName: \"kubernetes.io/projected/c775b400-2625-4656-a442-a8069e8bd20e-kube-api-access-rq958\") pod \"tuned-gklml\" (UID: \"c775b400-2625-4656-a442-a8069e8bd20e\") " pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.920118    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5znf6\" (UniqueName: \"kubernetes.io/projected/4173f069-b400-4be2-a30f-99c0d6b077cf-kube-api-access-5znf6\") pod \"aws-ebs-csi-driver-node-vzr6w\" (UID: \"4173f069-b400-4be2-a30f-99c0d6b077cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.925910    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qmr\" (UniqueName: \"kubernetes.io/projected/0723c42e-0fc4-4c03-b5e6-83d0af6a9488-kube-api-access-88qmr\") pod \"node-resolver-gd8x7\" (UID: \"0723c42e-0fc4-4c03-b5e6-83d0af6a9488\") " pod="openshift-dns/node-resolver-gd8x7"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.926899    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxkh\" (UniqueName: \"kubernetes.io/projected/a0e28d47-acbf-4c6b-ad83-a883ad149185-kube-api-access-htxkh\") pod \"node-ca-nrf8m\" (UID: \"a0e28d47-acbf-4c6b-ad83-a883ad149185\") " pod="openshift-image-registry/node-ca-nrf8m"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.927828    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhd4\" (UniqueName: \"kubernetes.io/projected/2bc24c26-f688-41bb-ad3e-27d184fe1669-kube-api-access-wwhd4\") pod \"iptables-alerter-ktjxz\" (UID: \"2bc24c26-f688-41bb-ad3e-27d184fe1669\") " pod="openshift-network-operator/iptables-alerter-ktjxz"
Jul 01 19:21:26.968686 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.928208    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25fkp\" (UniqueName: \"kubernetes.io/projected/908385ef-e1e8-4672-90f9-ef5f5a47f6aa-kube-api-access-25fkp\") pod \"multus-52gwt\" (UID: \"908385ef-e1e8-4672-90f9-ef5f5a47f6aa\") " pod="openshift-multus/multus-52gwt"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:26.984010    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-52gwt"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.018823    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-system-cni-dir\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.018855    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-systemd-units\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.018871    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-cni-netd\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.018888    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-var-lib-openvswitch\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.018905    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wr7g\" (UniqueName: \"kubernetes.io/projected/d69d7279-1771-42d1-bde5-3dbc8ee7f990-kube-api-access-5wr7g\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.018938    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-system-cni-dir\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.018950    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-systemd-units\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.018961    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-var-lib-openvswitch\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.018955    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-cni-netd\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.018997    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-run-systemd\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019024    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019042    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d69d7279-1771-42d1-bde5-3dbc8ee7f990-env-overrides\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019068    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-run-systemd\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.146297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019081    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-log-socket\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019120    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5a828de-7dd6-45af-b69c-1837becfe480-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019126    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019141    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-log-socket\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019144    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-node-log\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019183    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-node-log\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019250    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-run-netns\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019309    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-run-netns\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019329    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5a828de-7dd6-45af-b69c-1837becfe480-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019405    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-slash\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019435    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-cni-bin\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019462    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-cnibin\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019504    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz5s\" (UniqueName: \"kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s\") pod \"network-check-target-bc752\" (UID: \"538fff2a-3657-456b-9158-297890173d5c\") " pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019521    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-slash\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.148874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019532    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d69d7279-1771-42d1-bde5-3dbc8ee7f990-ovn-node-metrics-cert\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019548    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d69d7279-1771-42d1-bde5-3dbc8ee7f990-env-overrides\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019561    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfwdb\" (UniqueName: \"kubernetes.io/projected/b5a828de-7dd6-45af-b69c-1837becfe480-kube-api-access-gfwdb\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019550    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-cni-bin\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019606    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-kubelet\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019637    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-cnibin\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019679    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-kubelet\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019681    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-run-openvswitch\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019687    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5a828de-7dd6-45af-b69c-1837becfe480-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019726    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-run-openvswitch\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019838    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019871    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d69d7279-1771-42d1-bde5-3dbc8ee7f990-ovnkube-config\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019887    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-os-release\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019912    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-run-ovn\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.149554 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019929    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-etc-openvswitch\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019946    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-run-ovn-kubernetes\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019962    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5a828de-7dd6-45af-b69c-1837becfe480-cni-binary-copy\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.019986    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d69d7279-1771-42d1-bde5-3dbc8ee7f990-ovnkube-script-lib\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.020010    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-run-ovn\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.020025    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.020025    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-host-run-ovn-kubernetes\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.020038    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5a828de-7dd6-45af-b69c-1837becfe480-os-release\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.020077    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d69d7279-1771-42d1-bde5-3dbc8ee7f990-etc-openvswitch\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.020366    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5a828de-7dd6-45af-b69c-1837becfe480-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.020447    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5a828de-7dd6-45af-b69c-1837becfe480-cni-binary-copy\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.020448    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d69d7279-1771-42d1-bde5-3dbc8ee7f990-ovnkube-config\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.020589    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d69d7279-1771-42d1-bde5-3dbc8ee7f990-ovnkube-script-lib\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.021952    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d69d7279-1771-42d1-bde5-3dbc8ee7f990-ovn-node-metrics-cert\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.027916    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:27.150085 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.027946    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:27.150903 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.027961    2586 projected.go:196] Error preparing data for projected volume kube-api-access-6jz5s for pod openshift-network-diagnostics/network-check-target-bc752: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:27.150903 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.028030    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s podName:538fff2a-3657-456b-9158-297890173d5c nodeName:}" failed. No retries permitted until 2026-07-01 19:21:27.52800936 +0000 UTC m=+2.074258693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6jz5s" (UniqueName: "kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s") pod "network-check-target-bc752" (UID: "538fff2a-3657-456b-9158-297890173d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:27.150903 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.030590    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfwdb\" (UniqueName: \"kubernetes.io/projected/b5a828de-7dd6-45af-b69c-1837becfe480-kube-api-access-gfwdb\") pod \"multus-additional-cni-plugins-b29mt\" (UID: \"b5a828de-7dd6-45af-b69c-1837becfe480\") " pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.150903 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.031035    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wr7g\" (UniqueName: \"kubernetes.io/projected/d69d7279-1771-42d1-bde5-3dbc8ee7f990-kube-api-access-5wr7g\") pod \"ovnkube-node-bxjj5\" (UID: \"d69d7279-1771-42d1-bde5-3dbc8ee7f990\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.150903 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.118566    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:27.150903 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:27.124274    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea467cf493fb8782e204574fc06f1c8.slice/crio-da3ebd8940e0d0c3a7348bab7ba8f5ece5dba128dc71cec82b15b8dd5e44775f WatchSource:0}: Error finding container da3ebd8940e0d0c3a7348bab7ba8f5ece5dba128dc71cec82b15b8dd5e44775f: Status 404 returned error can't find the container with id da3ebd8940e0d0c3a7348bab7ba8f5ece5dba128dc71cec82b15b8dd5e44775f
Jul 01 19:21:27.150903 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:27.124753    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59f55115b683b9b149f1e104e483009.slice/crio-42258a51ee31795c37293c214d6b2628c7fbb63a80dca430847fa788e6c442e0 WatchSource:0}: Error finding container 42258a51ee31795c37293c214d6b2628c7fbb63a80dca430847fa788e6c442e0: Status 404 returned error can't find the container with id 42258a51ee31795c37293c214d6b2628c7fbb63a80dca430847fa788e6c442e0
Jul 01 19:21:27.150903 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.132189    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:21:27.150903 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.132398    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w"
Jul 01 19:21:27.150903 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:27.140679    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4173f069_b400_4be2_a30f_99c0d6b077cf.slice/crio-8e7d46cac0fc981fc86dcef478e4c7fb24b87ec40758aee43cceb76cbd30c6a4 WatchSource:0}: Error finding container 8e7d46cac0fc981fc86dcef478e4c7fb24b87ec40758aee43cceb76cbd30c6a4: Status 404 returned error can't find the container with id 8e7d46cac0fc981fc86dcef478e4c7fb24b87ec40758aee43cceb76cbd30c6a4
Jul 01 19:21:27.150903 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.148328    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.152824    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gklml"
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:27.162755    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc775b400_2625_4656_a442_a8069e8bd20e.slice/crio-b818f1efc5061a2513a1f7e400c69d08ff1dbf22b94c4e0e7ca99610cf5e07db WatchSource:0}: Error finding container b818f1efc5061a2513a1f7e400c69d08ff1dbf22b94c4e0e7ca99610cf5e07db: Status 404 returned error can't find the container with id b818f1efc5061a2513a1f7e400c69d08ff1dbf22b94c4e0e7ca99610cf5e07db
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.167642    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gd8x7"
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:27.175334    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0723c42e_0fc4_4c03_b5e6_83d0af6a9488.slice/crio-221ac3e908b8128907997680e7dd9275f6eea16b52e3effba4613381b443af8d WatchSource:0}: Error finding container 221ac3e908b8128907997680e7dd9275f6eea16b52e3effba4613381b443af8d: Status 404 returned error can't find the container with id 221ac3e908b8128907997680e7dd9275f6eea16b52e3effba4613381b443af8d
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.186310    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrf8m"
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:27.193252    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e28d47_acbf_4c6b_ad83_a883ad149185.slice/crio-21ed8acfa52ce1c668161e8e26ae18dd1722edea6be5873e51d270ae091abff3 WatchSource:0}: Error finding container 21ed8acfa52ce1c668161e8e26ae18dd1722edea6be5873e51d270ae091abff3: Status 404 returned error can't find the container with id 21ed8acfa52ce1c668161e8e26ae18dd1722edea6be5873e51d270ae091abff3
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.217792    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ktjxz"
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:27.224892    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc24c26_f688_41bb_ad3e_27d184fe1669.slice/crio-fceb27030767501235efaf0e0a83654ca3d9b6fd1e099f1276935ff57e70d1b6 WatchSource:0}: Error finding container fceb27030767501235efaf0e0a83654ca3d9b6fd1e099f1276935ff57e70d1b6: Status 404 returned error can't find the container with id fceb27030767501235efaf0e0a83654ca3d9b6fd1e099f1276935ff57e70d1b6
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.300785    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b29mt"
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.306135    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:27.308422    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a828de_7dd6_45af_b69c_1837becfe480.slice/crio-bbf9c164eb08a929d406e74b231e50a2c8caa99b320d45e0d889eb95d210490b WatchSource:0}: Error finding container bbf9c164eb08a929d406e74b231e50a2c8caa99b320d45e0d889eb95d210490b: Status 404 returned error can't find the container with id bbf9c164eb08a929d406e74b231e50a2c8caa99b320d45e0d889eb95d210490b
Jul 01 19:21:27.373538 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:27.314049    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69d7279_1771_42d1_bde5_3dbc8ee7f990.slice/crio-9356fe94eaec186fc7c83515137314d923a48f009db1ddeefbadf3d65b51bb6e WatchSource:0}: Error finding container 9356fe94eaec186fc7c83515137314d923a48f009db1ddeefbadf3d65b51bb6e: Status 404 returned error can't find the container with id 9356fe94eaec186fc7c83515137314d923a48f009db1ddeefbadf3d65b51bb6e
Jul 01 19:21:27.573600 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.402517    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:27.573600 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.424160    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:27.573600 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.424194    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:27.573600 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.424302    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:27.573600 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.424358    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret podName:55e6ffec-a820-4802-89cb-747770b45bb9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:28.424343588 +0000 UTC m=+2.970592905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret") pod "global-pull-secret-syncer-5zvjf" (UID: "55e6ffec-a820-4802-89cb-747770b45bb9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:27.573600 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.424303    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:27.573600 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.424421    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs podName:f41077b2-239b-44f8-be7d-b8b9464b42e9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:28.424408663 +0000 UTC m=+2.970657982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs") pod "network-metrics-daemon-jbvd8" (UID: "f41077b2-239b-44f8-be7d-b8b9464b42e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:27.573600 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:21:27.518656    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908385ef_e1e8_4672_90f9_ef5f5a47f6aa.slice/crio-1d4108894dc3f2da4bf32caea350705552b756a3b6b5d4726fbb07c5cb80b173 WatchSource:0}: Error finding container 1d4108894dc3f2da4bf32caea350705552b756a3b6b5d4726fbb07c5cb80b173: Status 404 returned error can't find the container with id 1d4108894dc3f2da4bf32caea350705552b756a3b6b5d4726fbb07c5cb80b173
Jul 01 19:21:27.656315 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.625802    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz5s\" (UniqueName: \"kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s\") pod \"network-check-target-bc752\" (UID: \"538fff2a-3657-456b-9158-297890173d5c\") " pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:27.656315 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.625969    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:27.656315 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.625990    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:27.656315 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.626004    2586 projected.go:196] Error preparing data for projected volume kube-api-access-6jz5s for pod openshift-network-diagnostics/network-check-target-bc752: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:27.656315 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:27.626070    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s podName:538fff2a-3657-456b-9158-297890173d5c nodeName:}" failed. No retries permitted until 2026-07-01 19:21:28.626051407 +0000 UTC m=+3.172300745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jz5s" (UniqueName: "kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s") pod "network-check-target-bc752" (UID: "538fff2a-3657-456b-9158-297890173d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:27.904197 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.881115    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:16:27 +0000 UTC" deadline="2028-01-08 07:18:08.095939014 +0000 UTC"
Jul 01 19:21:27.904197 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.881165    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13331h56m40.214778401s"
Jul 01 19:21:27.985458 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.908645    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-52gwt" event={"ID":"908385ef-e1e8-4672-90f9-ef5f5a47f6aa","Type":"ContainerStarted","Data":"1d4108894dc3f2da4bf32caea350705552b756a3b6b5d4726fbb07c5cb80b173"}
Jul 01 19:21:27.985458 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.929103    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5" event={"ID":"d69d7279-1771-42d1-bde5-3dbc8ee7f990","Type":"ContainerStarted","Data":"9356fe94eaec186fc7c83515137314d923a48f009db1ddeefbadf3d65b51bb6e"}
Jul 01 19:21:27.985458 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.948234    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b29mt" event={"ID":"b5a828de-7dd6-45af-b69c-1837becfe480","Type":"ContainerStarted","Data":"bbf9c164eb08a929d406e74b231e50a2c8caa99b320d45e0d889eb95d210490b"}
Jul 01 19:21:27.985458 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.966438    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ktjxz" event={"ID":"2bc24c26-f688-41bb-ad3e-27d184fe1669","Type":"ContainerStarted","Data":"fceb27030767501235efaf0e0a83654ca3d9b6fd1e099f1276935ff57e70d1b6"}
Jul 01 19:21:27.985458 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.981371    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrf8m" event={"ID":"a0e28d47-acbf-4c6b-ad83-a883ad149185","Type":"ContainerStarted","Data":"21ed8acfa52ce1c668161e8e26ae18dd1722edea6be5873e51d270ae091abff3"}
Jul 01 19:21:27.987425 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:27.985863    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w" event={"ID":"4173f069-b400-4be2-a30f-99c0d6b077cf","Type":"ContainerStarted","Data":"8e7d46cac0fc981fc86dcef478e4c7fb24b87ec40758aee43cceb76cbd30c6a4"}
Jul 01 19:21:28.006575 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.006537    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-56r4t" event={"ID":"9dd6796f-9e3d-4c78-93ec-acd06d798d3c","Type":"ContainerStarted","Data":"c58098d7874d60038db27ee93403059afe60d148dd12e6cdf4d9e0b19b153e6c"}
Jul 01 19:21:28.023992 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.023941    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.ec2.internal" event={"ID":"4ea467cf493fb8782e204574fc06f1c8","Type":"ContainerStarted","Data":"da3ebd8940e0d0c3a7348bab7ba8f5ece5dba128dc71cec82b15b8dd5e44775f"}
Jul 01 19:21:28.042472 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.042402    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gd8x7" event={"ID":"0723c42e-0fc4-4c03-b5e6-83d0af6a9488","Type":"ContainerStarted","Data":"221ac3e908b8128907997680e7dd9275f6eea16b52e3effba4613381b443af8d"}
Jul 01 19:21:28.051053 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.050957    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gklml" event={"ID":"c775b400-2625-4656-a442-a8069e8bd20e","Type":"ContainerStarted","Data":"b818f1efc5061a2513a1f7e400c69d08ff1dbf22b94c4e0e7ca99610cf5e07db"}
Jul 01 19:21:28.077084 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.076999    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal" event={"ID":"f59f55115b683b9b149f1e104e483009","Type":"ContainerStarted","Data":"42258a51ee31795c37293c214d6b2628c7fbb63a80dca430847fa788e6c442e0"}
Jul 01 19:21:28.272978 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.272944    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:28.436914 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.436817    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:28.436914 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.436869    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:28.437152 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:28.437042    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:28.437152 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:28.437104    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret podName:55e6ffec-a820-4802-89cb-747770b45bb9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:30.43708602 +0000 UTC m=+4.983335354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret") pod "global-pull-secret-syncer-5zvjf" (UID: "55e6ffec-a820-4802-89cb-747770b45bb9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:28.437673 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:28.437560    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:28.437673 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:28.437613    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs podName:f41077b2-239b-44f8-be7d-b8b9464b42e9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:30.437597076 +0000 UTC m=+4.983846406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs") pod "network-metrics-daemon-jbvd8" (UID: "f41077b2-239b-44f8-be7d-b8b9464b42e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:28.638934 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.638892    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz5s\" (UniqueName: \"kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s\") pod \"network-check-target-bc752\" (UID: \"538fff2a-3657-456b-9158-297890173d5c\") " pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:28.639155 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:28.639137    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:28.639224 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:28.639168    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:28.639224 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:28.639182    2586 projected.go:196] Error preparing data for projected volume kube-api-access-6jz5s for pod openshift-network-diagnostics/network-check-target-bc752: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:28.639339 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:28.639242    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s podName:538fff2a-3657-456b-9158-297890173d5c nodeName:}" failed. No retries permitted until 2026-07-01 19:21:30.639223089 +0000 UTC m=+5.185472428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jz5s" (UniqueName: "kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s") pod "network-check-target-bc752" (UID: "538fff2a-3657-456b-9158-297890173d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:28.878044 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.877957    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:28.878221 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:28.878097    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:28.878837 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.878537    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:28.878837 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:28.878654    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:28.878837 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.878719    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:28.878837 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:28.878809    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:28.882425 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.882033    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:16:27 +0000 UTC" deadline="2028-01-20 18:32:16.879584507 +0000 UTC"
Jul 01 19:21:28.882425 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:28.882063    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13631h10m47.997526428s"
Jul 01 19:21:30.459291 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:30.458394    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:30.459291 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:30.458459    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:30.459291 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:30.458652    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:30.459291 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:30.458719    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret podName:55e6ffec-a820-4802-89cb-747770b45bb9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:34.458700644 +0000 UTC m=+9.004949986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret") pod "global-pull-secret-syncer-5zvjf" (UID: "55e6ffec-a820-4802-89cb-747770b45bb9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:30.459291 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:30.459188    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:30.459291 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:30.459248    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs podName:f41077b2-239b-44f8-be7d-b8b9464b42e9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:34.459229562 +0000 UTC m=+9.005478897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs") pod "network-metrics-daemon-jbvd8" (UID: "f41077b2-239b-44f8-be7d-b8b9464b42e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:30.660210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:30.660164    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz5s\" (UniqueName: \"kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s\") pod \"network-check-target-bc752\" (UID: \"538fff2a-3657-456b-9158-297890173d5c\") " pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:30.660464 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:30.660441    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:30.660553 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:30.660471    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:30.660553 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:30.660484    2586 projected.go:196] Error preparing data for projected volume kube-api-access-6jz5s for pod openshift-network-diagnostics/network-check-target-bc752: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:30.660553 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:30.660552    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s podName:538fff2a-3657-456b-9158-297890173d5c nodeName:}" failed. No retries permitted until 2026-07-01 19:21:34.660530933 +0000 UTC m=+9.206780274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jz5s" (UniqueName: "kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s") pod "network-check-target-bc752" (UID: "538fff2a-3657-456b-9158-297890173d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:30.878508 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:30.878382    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:30.878666 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:30.878528    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:30.879109 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:30.878789    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:30.879109 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:30.878870    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:30.879109 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:30.878985    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:30.879109 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:30.879064    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:32.878167 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:32.878130    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:32.878673 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:32.878130    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:32.878673 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:32.878286    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:32.878673 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:32.878141    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:32.878673 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:32.878365    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:32.878673 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:32.878455    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:34.499887 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:34.499837    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:34.499887 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:34.499888    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:34.500511 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:34.500016    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:34.500511 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:34.500089    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs podName:f41077b2-239b-44f8-be7d-b8b9464b42e9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:42.500070944 +0000 UTC m=+17.046320258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs") pod "network-metrics-daemon-jbvd8" (UID: "f41077b2-239b-44f8-be7d-b8b9464b42e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:34.500511 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:34.500020    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:34.500511 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:34.500196    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret podName:55e6ffec-a820-4802-89cb-747770b45bb9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:42.500182906 +0000 UTC m=+17.046432238 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret") pod "global-pull-secret-syncer-5zvjf" (UID: "55e6ffec-a820-4802-89cb-747770b45bb9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:34.701353 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:34.701317    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz5s\" (UniqueName: \"kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s\") pod \"network-check-target-bc752\" (UID: \"538fff2a-3657-456b-9158-297890173d5c\") " pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:34.701536 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:34.701488    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:34.701536 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:34.701515    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:34.701536 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:34.701528    2586 projected.go:196] Error preparing data for projected volume kube-api-access-6jz5s for pod openshift-network-diagnostics/network-check-target-bc752: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:34.701713 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:34.701591    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s podName:538fff2a-3657-456b-9158-297890173d5c nodeName:}" failed. No retries permitted until 2026-07-01 19:21:42.701573534 +0000 UTC m=+17.247822850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jz5s" (UniqueName: "kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s") pod "network-check-target-bc752" (UID: "538fff2a-3657-456b-9158-297890173d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:34.879040 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:34.878964    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:34.879040 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:34.878999    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:34.879251 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:34.879098    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:34.879251 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:34.879150    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:34.879251 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:34.879222    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:34.879403 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:34.879322    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:36.878713 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:36.878334    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:36.878713 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:36.878442    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:36.878713 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:36.878335    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:36.878713 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:36.878504    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:36.878713 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:36.878335    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:36.878713 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:36.878549    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:38.878194 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:38.878156    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:38.878669 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:38.878163    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:38.878669 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:38.878280    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:38.878669 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:38.878276    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:38.878669 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:38.878354    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:38.878669 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:38.878412    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:40.878254 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:40.878231    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:40.878701 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:40.878265    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:40.878701 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:40.878326    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:40.878701 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:40.878401    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:40.878701 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:40.878454    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:40.878701 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:40.878563    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:42.565471 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:42.565439    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:42.565953 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:42.565481    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:42.565953 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:42.565610    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:42.565953 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:42.565632    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:42.565953 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:42.565673    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret podName:55e6ffec-a820-4802-89cb-747770b45bb9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:58.565654731 +0000 UTC m=+33.111904048 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret") pod "global-pull-secret-syncer-5zvjf" (UID: "55e6ffec-a820-4802-89cb-747770b45bb9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:42.565953 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:42.565708    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs podName:f41077b2-239b-44f8-be7d-b8b9464b42e9 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:58.565697374 +0000 UTC m=+33.111946692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs") pod "network-metrics-daemon-jbvd8" (UID: "f41077b2-239b-44f8-be7d-b8b9464b42e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:42.767593 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:42.767557    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz5s\" (UniqueName: \"kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s\") pod \"network-check-target-bc752\" (UID: \"538fff2a-3657-456b-9158-297890173d5c\") " pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:42.767782 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:42.767731    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:42.767782 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:42.767751    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:42.767782 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:42.767761    2586 projected.go:196] Error preparing data for projected volume kube-api-access-6jz5s for pod openshift-network-diagnostics/network-check-target-bc752: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:42.767922 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:42.767818    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s podName:538fff2a-3657-456b-9158-297890173d5c nodeName:}" failed. No retries permitted until 2026-07-01 19:21:58.76780041 +0000 UTC m=+33.314049800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jz5s" (UniqueName: "kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s") pod "network-check-target-bc752" (UID: "538fff2a-3657-456b-9158-297890173d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:42.878742 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:42.878643    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:42.878742 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:42.878695    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:42.878742 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:42.878678    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:42.878974 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:42.878810    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:42.878974 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:42.878891    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:42.879067 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:42.878972    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:44.878368 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:44.878319    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:44.878368 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:44.878327    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:44.878899 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:44.878319    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:44.878899 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:44.878466    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:44.878899 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:44.878586    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:44.878899 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:44.878677    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:46.878378 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:46.878346    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:46.878378 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:46.878385    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:46.878868 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:46.878464    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:46.878868 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:46.878475    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:46.878868 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:46.878556    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:46.878868 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:46.878639    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:48.878544 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:48.878211    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:48.879083 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:48.878205    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:48.879083 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:48.878595    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:48.879083 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:48.878238    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:48.879083 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:48.878663    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:48.879083 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:48.878771    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:49.149423 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.149395    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrf8m" event={"ID":"a0e28d47-acbf-4c6b-ad83-a883ad149185","Type":"ContainerStarted","Data":"d105fd1573afc9f5ddfe98d9ba76e6953ca146f4f6250861b506ce62ab09dc94"}
Jul 01 19:21:49.151882 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.151859    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w" event={"ID":"4173f069-b400-4be2-a30f-99c0d6b077cf","Type":"ContainerStarted","Data":"3b5e9dd36e69fad8fa6faaa130be2eac429920069300b6dd610646408afb688e"}
Jul 01 19:21:49.155112 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.155091    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-56r4t" event={"ID":"9dd6796f-9e3d-4c78-93ec-acd06d798d3c","Type":"ContainerStarted","Data":"b4d9f48dfae3139b80a5b1166aae1b70b341ee3eea02469ff694180c4c7a8861"}
Jul 01 19:21:49.156693 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.156657    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.ec2.internal" event={"ID":"4ea467cf493fb8782e204574fc06f1c8","Type":"ContainerStarted","Data":"af0d37d6c80349f3be5b571c5e87c5a2948114c2b275bddfd4c8ca2052b85804"}
Jul 01 19:21:49.158676 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.158626    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gd8x7" event={"ID":"0723c42e-0fc4-4c03-b5e6-83d0af6a9488","Type":"ContainerStarted","Data":"d81b858802bab60651ee7a01a80685fa4ef977ffa600e1c276f269bdb5a1fa8f"}
Jul 01 19:21:49.160402 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.160377    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gklml" event={"ID":"c775b400-2625-4656-a442-a8069e8bd20e","Type":"ContainerStarted","Data":"f8c4d59037ad46f60fd49325d19419b7ce001e369eab1cbd238b7f7bce0b919f"}
Jul 01 19:21:49.161737 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.161717    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal" event={"ID":"f59f55115b683b9b149f1e104e483009","Type":"ContainerStarted","Data":"cd3aec89386c92895d3e084cccf9423ade31b97c12713c83f8d6c39d900e26ae"}
Jul 01 19:21:49.163345 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.163327    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-52gwt" event={"ID":"908385ef-e1e8-4672-90f9-ef5f5a47f6aa","Type":"ContainerStarted","Data":"ca60ed6e9955d2239295c0c72dc83df57aef35abfde9a1ccc320ba5892c7afaf"}
Jul 01 19:21:49.165286 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.165231    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nrf8m" podStartSLOduration=7.050635509 podStartE2EDuration="23.165214796s" totalImagesPullingTime="16.114579287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:49.164477822 +0000 UTC m=+23.710727158" watchObservedRunningTime="2026-07-01 19:21:49.165214796 +0000 UTC m=+23.711464134"
Jul 01 19:21:49.166437 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.166417    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5" event={"ID":"d69d7279-1771-42d1-bde5-3dbc8ee7f990","Type":"ContainerStarted","Data":"b2c53ef13d0f678f5019d3d4a0fc5060cb1338c4a3cb2c48b2dd25c9fba588ee"}
Jul 01 19:21:49.166535 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.166488    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5" event={"ID":"d69d7279-1771-42d1-bde5-3dbc8ee7f990","Type":"ContainerStarted","Data":"c6cd6cf1dd217bf496d9a4b0945940ab4d9d3c5052e76e94799b93095bec5150"}
Jul 01 19:21:49.166597 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.166541    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5" event={"ID":"d69d7279-1771-42d1-bde5-3dbc8ee7f990","Type":"ContainerStarted","Data":"11a9ef85b4c9d115b12d63b6cdc2f1d1e30fff0a4ab1d852b76bf981b7753d64"}
Jul 01 19:21:49.166652 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.166619    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5" event={"ID":"d69d7279-1771-42d1-bde5-3dbc8ee7f990","Type":"ContainerStarted","Data":"8e06b57de9291d8321510b27ed594347b9d87d62dc0edb383bea7b425e648fc1"}
Jul 01 19:21:49.166738 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.166670    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5" event={"ID":"d69d7279-1771-42d1-bde5-3dbc8ee7f990","Type":"ContainerStarted","Data":"8d8e493b07ed9a6988c5209a45bfaa48776bfc3a139ee232467c7c6b25590d24"}
Jul 01 19:21:49.166738 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.166733    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5" event={"ID":"d69d7279-1771-42d1-bde5-3dbc8ee7f990","Type":"ContainerStarted","Data":"fe3a253ea5205d31325604576c4d55464028ba13e91e5b083d7dbf81f0811ecf"}
Jul 01 19:21:49.168103 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.168084    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b29mt" event={"ID":"b5a828de-7dd6-45af-b69c-1837becfe480","Type":"ContainerStarted","Data":"a8f0908253f60900eae13c8dcfd8ab45cdeac679731d6551ca8fbbd72eec6a66"}
Jul 01 19:21:49.182508 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.182470    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-52gwt" podStartSLOduration=2.670470558 podStartE2EDuration="23.182457741s" totalImagesPullingTime="20.511987183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:49.182031139 +0000 UTC m=+23.728280476" watchObservedRunningTime="2026-07-01 19:21:49.182457741 +0000 UTC m=+23.728707076"
Jul 01 19:21:49.218159 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.218118    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-93.ec2.internal" podStartSLOduration=22.218104302 podStartE2EDuration="22.218104302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:49.200303663 +0000 UTC m=+23.746552999" watchObservedRunningTime="2026-07-01 19:21:49.218104302 +0000 UTC m=+23.764353712"
Jul 01 19:21:49.218333 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.218295    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gd8x7" podStartSLOduration=2.358358117 podStartE2EDuration="23.218290609s" totalImagesPullingTime="20.859932492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:49.217810082 +0000 UTC m=+23.764059418" watchObservedRunningTime="2026-07-01 19:21:49.218290609 +0000 UTC m=+23.764539945"
Jul 01 19:21:49.250603 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.250561    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-56r4t" podStartSLOduration=2.4151127 podStartE2EDuration="23.250547264s" totalImagesPullingTime="20.835434564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:49.249962535 +0000 UTC m=+23.796211871" watchObservedRunningTime="2026-07-01 19:21:49.250547264 +0000 UTC m=+23.796796599"
Jul 01 19:21:49.269530 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.269491    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gklml" podStartSLOduration=2.401999731 podStartE2EDuration="23.269478033s" totalImagesPullingTime="20.867478302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:49.269116784 +0000 UTC m=+23.815366121" watchObservedRunningTime="2026-07-01 19:21:49.269478033 +0000 UTC m=+23.815727369"
Jul 01 19:21:49.353829 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.353793    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:49.420662 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.420583    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:49.938622 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.938591    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:49.939375 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:49.939262    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:50.171903 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.171866    2586 generic.go:356] "Generic (PLEG): container finished" podID="f59f55115b683b9b149f1e104e483009" containerID="cd3aec89386c92895d3e084cccf9423ade31b97c12713c83f8d6c39d900e26ae" exitCode=0
Jul 01 19:21:50.172080 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.171959    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal" event={"ID":"f59f55115b683b9b149f1e104e483009","Type":"ContainerDied","Data":"cd3aec89386c92895d3e084cccf9423ade31b97c12713c83f8d6c39d900e26ae"}
Jul 01 19:21:50.173777 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.173756    2586 generic.go:356] "Generic (PLEG): container finished" podID="b5a828de-7dd6-45af-b69c-1837becfe480" containerID="a8f0908253f60900eae13c8dcfd8ab45cdeac679731d6551ca8fbbd72eec6a66" exitCode=0
Jul 01 19:21:50.173885 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.173807    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b29mt" event={"ID":"b5a828de-7dd6-45af-b69c-1837becfe480","Type":"ContainerDied","Data":"a8f0908253f60900eae13c8dcfd8ab45cdeac679731d6551ca8fbbd72eec6a66"}
Jul 01 19:21:50.175489 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.175464    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ktjxz" event={"ID":"2bc24c26-f688-41bb-ad3e-27d184fe1669","Type":"ContainerStarted","Data":"90bb7efd9bf45a54e67cdcf09701369f03593f93ca6a5367f1da41fb12ae32cd"}
Jul 01 19:21:50.176498 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.176413    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:50.176869 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.176847    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-56r4t"
Jul 01 19:21:50.251052 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.250985    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:50.582923 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.582750    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:50.583433 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.583410    2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 19:21:50.684025 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.683990    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:50.878521 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.878445    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:50.878521 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.878476    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:50.878764 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.878565    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:50.878764 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:50.878568    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:50.878764 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:50.878635    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:50.878764 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:50.878747    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:50.879493 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.879410    2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T19:21:50.583425183Z","UUID":"13be9b2e-952d-415e-b8d7-7e1f0a1adcf7","Handler":null,"Name":"","Endpoint":""}
Jul 01 19:21:50.881047 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.881010    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 19:21:50.881047 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:50.881049    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 19:21:51.181114 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:51.181036    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5" event={"ID":"d69d7279-1771-42d1-bde5-3dbc8ee7f990","Type":"ContainerStarted","Data":"335202b9f48db303935ede9b303fc19b1555455c23b689bd37bc1ebde5485eaa"}
Jul 01 19:21:51.183076 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:51.183054    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w" event={"ID":"4173f069-b400-4be2-a30f-99c0d6b077cf","Type":"ContainerStarted","Data":"7ada6a1aa3dceddc7a679f9b2a45b8e0b49df52ae1b8da9ad16ad598f3d196b8"}
Jul 01 19:21:51.184943 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:51.184917    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal" event={"ID":"f59f55115b683b9b149f1e104e483009","Type":"ContainerStarted","Data":"eb88df11c49a0ed55c5a6b033bbbc421944ca1180f4d17450aa33ae8de2cfe2a"}
Jul 01 19:21:51.200364 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:51.200326    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-93.ec2.internal" podStartSLOduration=24.200299345 podStartE2EDuration="24.200299345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:51.199912957 +0000 UTC m=+25.746162292" watchObservedRunningTime="2026-07-01 19:21:51.200299345 +0000 UTC m=+25.746548681"
Jul 01 19:21:51.200670 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:51.200649    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ktjxz" podStartSLOduration=4.415560413 podStartE2EDuration="25.200644478s" totalImagesPullingTime="20.785084065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:50.238661646 +0000 UTC m=+24.784910982" watchObservedRunningTime="2026-07-01 19:21:51.200644478 +0000 UTC m=+25.746893815"
Jul 01 19:21:52.188465 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:52.188422    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:52.190015 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:52.189980    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w" event={"ID":"4173f069-b400-4be2-a30f-99c0d6b077cf","Type":"ContainerStarted","Data":"19557c20ee8bb1ac1b36d265c26d8ccb88a56b2a4409157782b0ba2788d9f196"}
Jul 01 19:21:52.210863 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:52.210821    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vzr6w" podStartSLOduration=2.618001673 podStartE2EDuration="26.210807348s" totalImagesPullingTime="23.592805675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:52.210153192 +0000 UTC m=+26.756402528" watchObservedRunningTime="2026-07-01 19:21:52.210807348 +0000 UTC m=+26.757056680"
Jul 01 19:21:52.289132 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:52.289039    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:52.410800 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:52.410757    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:52.483740 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:52.483704    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:52.878413 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:52.878381    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:52.878413 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:52.878417    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:52.878646 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:52.878498    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:52.878646 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:52.878502    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:52.878646 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:52.878600    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:52.878792 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:52.878712    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:53.196466 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:53.196130    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5" event={"ID":"d69d7279-1771-42d1-bde5-3dbc8ee7f990","Type":"ContainerStarted","Data":"20dbfac418450784c5202eb1f924d2ab0b876ec560d5dcd9a8c7fec7bc6f92ae"}
Jul 01 19:21:53.244470 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:53.244413    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5" podStartSLOduration=6.278744148 podStartE2EDuration="27.244398255s" totalImagesPullingTime="20.965654107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:53.24407046 +0000 UTC m=+27.790319801" watchObservedRunningTime="2026-07-01 19:21:53.244398255 +0000 UTC m=+27.790647591"
Jul 01 19:21:54.198981 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:54.198707    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:54.199453 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:54.198997    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:54.199453 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:54.199046    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:54.214568 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:54.214541    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:54.214770 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:54.214756    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:21:54.811160 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:54.811125    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:54.878208 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:54.878176    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:54.878392 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:54.878213    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:54.878392 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:54.878217    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:54.878392 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:54.878285    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:54.878392 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:54.878353    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:54.878637 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:54.878438    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:54.959296 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:54.959263    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:55.064944 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:55.064863    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:55.362039 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:55.361975    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5zvjf"]
Jul 01 19:21:55.362505 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:55.362061    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:55.362505 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:55.362139    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:55.365838 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:55.365813    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jbvd8"]
Jul 01 19:21:55.365957 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:55.365919    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:55.366055 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:55.366035    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:55.366329 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:55.366311    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bc752"]
Jul 01 19:21:55.366395 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:55.366384    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:55.366470 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:55.366455    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:56.878172 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:56.878136    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:56.878172 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:56.878173    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:56.878668 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:56.878176    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:56.878668 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:56.878273    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:56.878668 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:56.878364    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:56.878668 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:56.878443    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:58.584976 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:58.584805    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:58.584976 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:58.584958    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:58.584976 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:58.584979    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:58.585450 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:58.585028    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs podName:f41077b2-239b-44f8-be7d-b8b9464b42e9 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:30.585006465 +0000 UTC m=+65.131255793 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs") pod "network-metrics-daemon-jbvd8" (UID: "f41077b2-239b-44f8-be7d-b8b9464b42e9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:58.585450 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:58.585059    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:58.585450 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:58.585119    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret podName:55e6ffec-a820-4802-89cb-747770b45bb9 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:30.585088962 +0000 UTC m=+65.131338276 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret") pod "global-pull-secret-syncer-5zvjf" (UID: "55e6ffec-a820-4802-89cb-747770b45bb9") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:58.683607 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:58.683528    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:58.786487 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:58.786457    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz5s\" (UniqueName: \"kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s\") pod \"network-check-target-bc752\" (UID: \"538fff2a-3657-456b-9158-297890173d5c\") " pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:58.786649 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:58.786578    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:58.786649 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:58.786594    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:58.786649 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:58.786604    2586 projected.go:196] Error preparing data for projected volume kube-api-access-6jz5s for pod openshift-network-diagnostics/network-check-target-bc752: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:58.786649 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:58.786648    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s podName:538fff2a-3657-456b-9158-297890173d5c nodeName:}" failed. No retries permitted until 2026-07-01 19:22:30.786636196 +0000 UTC m=+65.332885509 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6jz5s" (UniqueName: "kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s") pod "network-check-target-bc752" (UID: "538fff2a-3657-456b-9158-297890173d5c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:58.812064 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:58.812029    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:58.878259 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:58.878230    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:21:58.878421 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:58.878230    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:21:58.878421 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:58.878326    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bc752" podUID="538fff2a-3657-456b-9158-297890173d5c"
Jul 01 19:21:58.878421 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:58.878230    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:21:58.878560 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:58.878394    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zvjf" podUID="55e6ffec-a820-4802-89cb-747770b45bb9"
Jul 01 19:21:58.878560 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:21:58.878507    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbvd8" podUID="f41077b2-239b-44f8-be7d-b8b9464b42e9"
Jul 01 19:21:59.211226 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:59.211188    2586 generic.go:356] "Generic (PLEG): container finished" podID="b5a828de-7dd6-45af-b69c-1837becfe480" containerID="c095c21d5ac4828a3b9aa4357e82ab074e14aebffcb57725c55f1730ce59994f" exitCode=0
Jul 01 19:21:59.211365 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:59.211233    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b29mt" event={"ID":"b5a828de-7dd6-45af-b69c-1837becfe480","Type":"ContainerDied","Data":"c095c21d5ac4828a3b9aa4357e82ab074e14aebffcb57725c55f1730ce59994f"}
Jul 01 19:21:59.456440 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:59.456398    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:21:59.561049 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:21:59.560964    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:00.215932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.215901    2586 generic.go:356] "Generic (PLEG): container finished" podID="b5a828de-7dd6-45af-b69c-1837becfe480" containerID="facf7153e00e1f0e74e566641df199e8a82cf6cb75939e8fb0735eefa9a3e235" exitCode=0
Jul 01 19:22:00.216433 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.215951    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b29mt" event={"ID":"b5a828de-7dd6-45af-b69c-1837becfe480","Type":"ContainerDied","Data":"facf7153e00e1f0e74e566641df199e8a82cf6cb75939e8fb0735eefa9a3e235"}
Jul 01 19:22:00.586817 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.586728    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:00.684128 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.684098    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:00.769187 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.769164    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-93.ec2.internal" event="NodeReady"
Jul 01 19:22:00.769349 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.769279    2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 19:22:00.838887 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.838811    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-566f556b75-9mp45"]
Jul 01 19:22:00.840730 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.840716    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:00.845781 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.845758    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 19:22:00.848619 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.848594    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jjqmq"]
Jul 01 19:22:00.850349 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.850335    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sdr2c"]
Jul 01 19:22:00.850456 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.850442    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jjqmq"
Jul 01 19:22:00.851865 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.851850    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:00.852442 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.852417    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 19:22:00.852707 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:00.852673    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-d8sfh\" is forbidden: User \"system:node:ip-10-0-141-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-141-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d8sfh\"" type="*v1.Secret"
Jul 01 19:22:00.853219 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:00.853194    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-93.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-141-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 01 19:22:00.853289 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.853212    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wklt5\""
Jul 01 19:22:00.853289 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:00.853236    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-93.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-141-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 01 19:22:00.853848 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.853826    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 19:22:00.854209 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:00.854183    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"dns-default\" is forbidden: User \"system:node:ip-10-0-141-93.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-141-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap"
Jul 01 19:22:00.854331 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:00.854310    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-dockercfg-f8prn\" is forbidden: User \"system:node:ip-10-0-141-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-141-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f8prn\"" type="*v1.Secret"
Jul 01 19:22:00.854744 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:00.854718    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"canary-serving-cert\" is forbidden: User \"system:node:ip-10-0-141-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-141-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret"
Jul 01 19:22:00.854744 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:00.854733    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-default-metrics-tls\" is forbidden: User \"system:node:ip-10-0-141-93.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-141-93.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret"
Jul 01 19:22:00.858042 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.858023    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 19:22:00.863514 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.863495    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jjqmq"]
Jul 01 19:22:00.864468 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.864446    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-566f556b75-9mp45"]
Jul 01 19:22:00.867939 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.867920    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sdr2c"]
Jul 01 19:22:00.878788 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.878764    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:22:00.878788 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.878777    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:22:00.878928 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.878763    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:22:00.884789 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.884770    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 19:22:00.884855 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.884807    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 19:22:00.885021 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.885007    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 19:22:00.885121 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.885107    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zbvbl\""
Jul 01 19:22:00.885237 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.885224    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mzk4c\""
Jul 01 19:22:00.885631 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.885598    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 19:22:00.976532 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:00.976490    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:01.006411 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006378    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-installation-pull-secrets\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.006582 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006492    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nhvz\" (UniqueName: \"kubernetes.io/projected/9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6-kube-api-access-8nhvz\") pod \"ingress-canary-jjqmq\" (UID: \"9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6\") " pod="openshift-ingress-canary/ingress-canary-jjqmq"
Jul 01 19:22:01.006582 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006526    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.006582 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006548    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-image-registry-private-configuration\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.006582 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006566    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-trusted-ca\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.006807 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006588    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60583eee-9b12-46e0-a65d-d9ca56efc9fb-tmp-dir\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:01.006807 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006672    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6-cert\") pod \"ingress-canary-jjqmq\" (UID: \"9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6\") " pod="openshift-ingress-canary/ingress-canary-jjqmq"
Jul 01 19:22:01.006807 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006743    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-bound-sa-token\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.006807 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006782    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:01.006948 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006807    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjk8\" (UniqueName: \"kubernetes.io/projected/60583eee-9b12-46e0-a65d-d9ca56efc9fb-kube-api-access-2wjk8\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:01.006948 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006872    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60583eee-9b12-46e0-a65d-d9ca56efc9fb-config-volume\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:01.006948 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006914    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dafde6e8-47b0-4b6c-aa45-c407774338c5-ca-trust-extracted\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.007050 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006963    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffkn8\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-kube-api-access-ffkn8\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.007050 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.006996    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-certificates\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.047883 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.047850    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:01.107868 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.107709    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60583eee-9b12-46e0-a65d-d9ca56efc9fb-config-volume\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:01.107868 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.107753    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dafde6e8-47b0-4b6c-aa45-c407774338c5-ca-trust-extracted\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.107868 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.107817    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffkn8\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-kube-api-access-ffkn8\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.107868 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.107859    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-certificates\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.108111 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.107887    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-installation-pull-secrets\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.108111 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.107943    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nhvz\" (UniqueName: \"kubernetes.io/projected/9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6-kube-api-access-8nhvz\") pod \"ingress-canary-jjqmq\" (UID: \"9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6\") " pod="openshift-ingress-canary/ingress-canary-jjqmq"
Jul 01 19:22:01.108111 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.107970    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.108111 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.108001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-image-registry-private-configuration\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.108111 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.108026    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-trusted-ca\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.108111 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.108053    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60583eee-9b12-46e0-a65d-d9ca56efc9fb-tmp-dir\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:01.108111 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.108099    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6-cert\") pod \"ingress-canary-jjqmq\" (UID: \"9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6\") " pod="openshift-ingress-canary/ingress-canary-jjqmq"
Jul 01 19:22:01.108408 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.108124    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-bound-sa-token\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.108408 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.108160    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:01.108408 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.108184    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjk8\" (UniqueName: \"kubernetes.io/projected/60583eee-9b12-46e0-a65d-d9ca56efc9fb-kube-api-access-2wjk8\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:01.108408 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:01.108313    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:01.108408 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:01.108330    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-566f556b75-9mp45: secret "image-registry-tls" not found
Jul 01 19:22:01.108642 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:01.108429    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls podName:dafde6e8-47b0-4b6c-aa45-c407774338c5 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:01.608377896 +0000 UTC m=+36.154627228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls") pod "image-registry-566f556b75-9mp45" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5") : secret "image-registry-tls" not found
Jul 01 19:22:01.108642 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.108570    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60583eee-9b12-46e0-a65d-d9ca56efc9fb-tmp-dir\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:01.109034 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.109008    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dafde6e8-47b0-4b6c-aa45-c407774338c5-ca-trust-extracted\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.109400 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.109313    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-trusted-ca\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.109501 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.109483    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-certificates\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.113479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.113456    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-installation-pull-secrets\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.113570 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.113482    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-image-registry-private-configuration\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.119330 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.119302    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-bound-sa-token\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.119452 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.119404    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjk8\" (UniqueName: \"kubernetes.io/projected/60583eee-9b12-46e0-a65d-d9ca56efc9fb-kube-api-access-2wjk8\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:01.119516 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.119473    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffkn8\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-kube-api-access-ffkn8\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.220473 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.220443    2586 generic.go:356] "Generic (PLEG): container finished" podID="b5a828de-7dd6-45af-b69c-1837becfe480" containerID="d8ab37bbdd1586d23c2a63bf781ec57fbfcd83ee86abedf4638a72f6fe15fdfd" exitCode=0
Jul 01 19:22:01.220777 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.220489    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b29mt" event={"ID":"b5a828de-7dd6-45af-b69c-1837becfe480","Type":"ContainerDied","Data":"d8ab37bbdd1586d23c2a63bf781ec57fbfcd83ee86abedf4638a72f6fe15fdfd"}
Jul 01 19:22:01.612318 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.612143    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:01.612457 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:01.612283    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:01.612457 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:01.612360    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-566f556b75-9mp45: secret "image-registry-tls" not found
Jul 01 19:22:01.612457 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:01.612412    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls podName:dafde6e8-47b0-4b6c-aa45-c407774338c5 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:02.612398584 +0000 UTC m=+37.158647898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls") pod "image-registry-566f556b75-9mp45" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5") : secret "image-registry-tls" not found
Jul 01 19:22:01.793786 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.793752    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 19:22:01.875103 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.875070    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 19:22:01.879848 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.879816    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60583eee-9b12-46e0-a65d-d9ca56efc9fb-config-volume\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:01.916478 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:01.916413    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 19:22:01.919388 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:01.919368    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:01.919494 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:01.919441    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls podName:60583eee-9b12-46e0-a65d-d9ca56efc9fb nodeName:}" failed. No retries permitted until 2026-07-01 19:22:02.419418442 +0000 UTC m=+36.965667768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls") pod "dns-default-sdr2c" (UID: "60583eee-9b12-46e0-a65d-d9ca56efc9fb") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:02.109087 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:02.109008    2586 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: failed to sync secret cache: timed out waiting for the condition
Jul 01 19:22:02.109248 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:02.109092    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6-cert podName:9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:02.609075623 +0000 UTC m=+37.155324937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6-cert") pod "ingress-canary-jjqmq" (UID: "9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6") : failed to sync secret cache: timed out waiting for the condition
Jul 01 19:22:02.332179 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:02.332144    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 19:22:02.340702 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:02.340663    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nhvz\" (UniqueName: \"kubernetes.io/projected/9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6-kube-api-access-8nhvz\") pod \"ingress-canary-jjqmq\" (UID: \"9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6\") " pod="openshift-ingress-canary/ingress-canary-jjqmq"
Jul 01 19:22:02.364608 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:02.364546    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 19:22:02.393381 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:02.393353    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f8prn\""
Jul 01 19:22:02.414297 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:02.414273    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d8sfh\""
Jul 01 19:22:02.420647 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:02.420620    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:02.420780 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:02.420746    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:02.420857 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:02.420847    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls podName:60583eee-9b12-46e0-a65d-d9ca56efc9fb nodeName:}" failed. No retries permitted until 2026-07-01 19:22:03.420833124 +0000 UTC m=+37.967082438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls") pod "dns-default-sdr2c" (UID: "60583eee-9b12-46e0-a65d-d9ca56efc9fb") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:02.622780 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:02.622709    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:02.622780 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:02.622772    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6-cert\") pod \"ingress-canary-jjqmq\" (UID: \"9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6\") " pod="openshift-ingress-canary/ingress-canary-jjqmq"
Jul 01 19:22:02.623031 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:02.622880    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:02.623031 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:02.622902    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-566f556b75-9mp45: secret "image-registry-tls" not found
Jul 01 19:22:02.623031 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:02.622967    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls podName:dafde6e8-47b0-4b6c-aa45-c407774338c5 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:04.622948118 +0000 UTC m=+39.169197433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls") pod "image-registry-566f556b75-9mp45" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5") : secret "image-registry-tls" not found
Jul 01 19:22:02.625558 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:02.625532    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6-cert\") pod \"ingress-canary-jjqmq\" (UID: \"9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6\") " pod="openshift-ingress-canary/ingress-canary-jjqmq"
Jul 01 19:22:02.659897 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:02.659868    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jjqmq"
Jul 01 19:22:02.841978 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:02.841947    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jjqmq"]
Jul 01 19:22:02.846024 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:22:02.845983    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1905ad_8654_45ed_b2d8_2f3cb60cdfd6.slice/crio-5b6625a9a3995a60f21f9c0d2a179b71b7d06d358db7f2a3a40c857309f2ce2d WatchSource:0}: Error finding container 5b6625a9a3995a60f21f9c0d2a179b71b7d06d358db7f2a3a40c857309f2ce2d: Status 404 returned error can't find the container with id 5b6625a9a3995a60f21f9c0d2a179b71b7d06d358db7f2a3a40c857309f2ce2d
Jul 01 19:22:03.227098 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:03.227061    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jjqmq" event={"ID":"9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6","Type":"ContainerStarted","Data":"5b6625a9a3995a60f21f9c0d2a179b71b7d06d358db7f2a3a40c857309f2ce2d"}
Jul 01 19:22:03.428650 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:03.428611    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:03.429444 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:03.428756    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:03.429444 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:03.428817    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls podName:60583eee-9b12-46e0-a65d-d9ca56efc9fb nodeName:}" failed. No retries permitted until 2026-07-01 19:22:05.428801312 +0000 UTC m=+39.975050631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls") pod "dns-default-sdr2c" (UID: "60583eee-9b12-46e0-a65d-d9ca56efc9fb") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:04.640704 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:04.640646    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:04.641307 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:04.640830    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:04.641307 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:04.640853    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-566f556b75-9mp45: secret "image-registry-tls" not found
Jul 01 19:22:04.641307 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:04.640926    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls podName:dafde6e8-47b0-4b6c-aa45-c407774338c5 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:08.640907792 +0000 UTC m=+43.187157125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls") pod "image-registry-566f556b75-9mp45" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5") : secret "image-registry-tls" not found
Jul 01 19:22:05.446571 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:05.446530    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:05.446777 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:05.446673    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:05.446777 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:05.446757    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls podName:60583eee-9b12-46e0-a65d-d9ca56efc9fb nodeName:}" failed. No retries permitted until 2026-07-01 19:22:09.446738947 +0000 UTC m=+43.992988261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls") pod "dns-default-sdr2c" (UID: "60583eee-9b12-46e0-a65d-d9ca56efc9fb") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:06.415198 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:06.415164    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:06.515856 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:06.515826    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:06.616467 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:06.616427    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:06.953103 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:06.953069    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:07.053166 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:07.053132    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:07.238555 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:07.238467    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jjqmq" event={"ID":"9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6","Type":"ContainerStarted","Data":"2423a1bca121ce8b6c35a0f894a61a3e73335d72920a0aa9f930ac582db4af35"}
Jul 01 19:22:07.241431 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:07.241405    2586 generic.go:356] "Generic (PLEG): container finished" podID="b5a828de-7dd6-45af-b69c-1837becfe480" containerID="4d7429e0145bdcfb5d95ec9067ec9948a105605b9d399579566fc2499fc28fb0" exitCode=0
Jul 01 19:22:07.241553 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:07.241448    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b29mt" event={"ID":"b5a828de-7dd6-45af-b69c-1837becfe480","Type":"ContainerDied","Data":"4d7429e0145bdcfb5d95ec9067ec9948a105605b9d399579566fc2499fc28fb0"}
Jul 01 19:22:07.277529 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:07.277480    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jjqmq" podStartSLOduration=3.714867205 podStartE2EDuration="7.277463125s" totalImagesPullingTime="3.56259592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:07.275729143 +0000 UTC m=+41.821978481" watchObservedRunningTime="2026-07-01 19:22:07.277463125 +0000 UTC m=+41.823712469"
Jul 01 19:22:07.331558 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:07.331532    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:07.433427 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:07.433394    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:07.783167 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:07.783134    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:07.881364 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:07.881326    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:07.979719 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:07.979669    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:08.236855 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.236826    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:08.245797 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.245768    2586 generic.go:356] "Generic (PLEG): container finished" podID="b5a828de-7dd6-45af-b69c-1837becfe480" containerID="1cbb2e744b49dd9e4ecd74a41e0555a624ca14b694888676bbe911a89fb329c1" exitCode=0
Jul 01 19:22:08.245936 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.245852    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b29mt" event={"ID":"b5a828de-7dd6-45af-b69c-1837becfe480","Type":"ContainerDied","Data":"1cbb2e744b49dd9e4ecd74a41e0555a624ca14b694888676bbe911a89fb329c1"}
Jul 01 19:22:08.312430 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.312386    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:08.431461 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.431416    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5hm57"]
Jul 01 19:22:08.448176 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.448150    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.451965 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.451940    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5hm57"]
Jul 01 19:22:08.452881 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.452863    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 19:22:08.452993 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.452916    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tz26f\""
Jul 01 19:22:08.453531 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.453518    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 19:22:08.453576 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.453547    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 19:22:08.454265 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.454086    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 19:22:08.469930 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.469903    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.470063 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.469958    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.470063 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.470016    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88c77ad8-3d00-4d21-8ac6-d6772afccf31-crio-socket\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.470063 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.470052    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c77ad8-3d00-4d21-8ac6-d6772afccf31-data-volume\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.470188 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.470078    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8w28\" (UniqueName: \"kubernetes.io/projected/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-api-access-q8w28\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.485355 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.485325    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:08.570573 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.570492    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.570573 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.570538    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88c77ad8-3d00-4d21-8ac6-d6772afccf31-crio-socket\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.570573 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.570562    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c77ad8-3d00-4d21-8ac6-d6772afccf31-data-volume\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.570823 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.570583    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8w28\" (UniqueName: \"kubernetes.io/projected/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-api-access-q8w28\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.570823 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.570630    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.570823 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:08.570659    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:08.570823 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:08.570728    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:22:08.570823 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.570750    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88c77ad8-3d00-4d21-8ac6-d6772afccf31-crio-socket\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.570823 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:08.570757    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls podName:88c77ad8-3d00-4d21-8ac6-d6772afccf31 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:09.070735849 +0000 UTC m=+43.616985163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hm57" (UID: "88c77ad8-3d00-4d21-8ac6-d6772afccf31") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:08.570823 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:08.570794    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm podName:88c77ad8-3d00-4d21-8ac6-d6772afccf31 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:09.070781347 +0000 UTC m=+43.617030665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5hm57" (UID: "88c77ad8-3d00-4d21-8ac6-d6772afccf31") : configmap "kube-rbac-proxy" not found
Jul 01 19:22:08.571042 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.570932    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88c77ad8-3d00-4d21-8ac6-d6772afccf31-data-volume\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.580163 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.580141    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8w28\" (UniqueName: \"kubernetes.io/projected/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-api-access-q8w28\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:08.602211 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.602182    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:08.671523 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:08.671487    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:08.671711 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:08.671638    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:08.671711 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:08.671651    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-566f556b75-9mp45: secret "image-registry-tls" not found
Jul 01 19:22:08.671794 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:08.671735    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls podName:dafde6e8-47b0-4b6c-aa45-c407774338c5 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:16.671717433 +0000 UTC m=+51.217966768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls") pod "image-registry-566f556b75-9mp45" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5") : secret "image-registry-tls" not found
Jul 01 19:22:09.074312 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:09.074274    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:09.074495 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:09.074329    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:09.074495 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:09.074409    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:22:09.074495 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:09.074431    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:09.074495 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:09.074475    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls podName:88c77ad8-3d00-4d21-8ac6-d6772afccf31 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:10.074459685 +0000 UTC m=+44.620709004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hm57" (UID: "88c77ad8-3d00-4d21-8ac6-d6772afccf31") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:09.074495 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:09.074487    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm podName:88c77ad8-3d00-4d21-8ac6-d6772afccf31 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:10.07448116 +0000 UTC m=+44.620730474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5hm57" (UID: "88c77ad8-3d00-4d21-8ac6-d6772afccf31") : configmap "kube-rbac-proxy" not found
Jul 01 19:22:09.251394 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:09.251360    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b29mt" event={"ID":"b5a828de-7dd6-45af-b69c-1837becfe480","Type":"ContainerStarted","Data":"c725a9f0ecad70d006956302bf0c9f80ccafbd49b284388f0dfe9a2e1604f265"}
Jul 01 19:22:09.282671 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:09.282627    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b29mt" podStartSLOduration=7.69137153 podStartE2EDuration="43.282610514s" totalImagesPullingTime="35.591238984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:09.281412151 +0000 UTC m=+43.827661487" watchObservedRunningTime="2026-07-01 19:22:09.282610514 +0000 UTC m=+43.828859849"
Jul 01 19:22:09.477202 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:09.477172    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:09.477653 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:09.477347    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:09.477653 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:09.477430    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls podName:60583eee-9b12-46e0-a65d-d9ca56efc9fb nodeName:}" failed. No retries permitted until 2026-07-01 19:22:17.477409521 +0000 UTC m=+52.023658850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls") pod "dns-default-sdr2c" (UID: "60583eee-9b12-46e0-a65d-d9ca56efc9fb") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:09.524064 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:09.524029    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:09.623415 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:09.623376    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:10.081097 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:10.081055    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:10.081294 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:10.081184    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:10.081294 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:10.081224    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:10.081294 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:10.081280    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:22:10.081427 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:10.081296    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls podName:88c77ad8-3d00-4d21-8ac6-d6772afccf31 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:12.081279061 +0000 UTC m=+46.627528377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hm57" (UID: "88c77ad8-3d00-4d21-8ac6-d6772afccf31") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:10.081427 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:10.081332    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm podName:88c77ad8-3d00-4d21-8ac6-d6772afccf31 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:12.081315894 +0000 UTC m=+46.627565209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5hm57" (UID: "88c77ad8-3d00-4d21-8ac6-d6772afccf31") : configmap "kube-rbac-proxy" not found
Jul 01 19:22:10.154052 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:10.154016    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:10.253789 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:10.253759    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:11.014526 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:11.014493    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gd8x7_0723c42e-0fc4-4c03-b5e6-83d0af6a9488/dns-node-resolver/0.log"
Jul 01 19:22:11.207929 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:11.207883    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:12.101501 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:12.101464    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:12.101905 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:12.101568    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:12.101905 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:12.101613    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:12.101905 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:12.101645    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:22:12.101905 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:12.101676    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls podName:88c77ad8-3d00-4d21-8ac6-d6772afccf31 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:16.101660389 +0000 UTC m=+50.647909706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hm57" (UID: "88c77ad8-3d00-4d21-8ac6-d6772afccf31") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:12.101905 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:12.101714    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm podName:88c77ad8-3d00-4d21-8ac6-d6772afccf31 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:16.101703367 +0000 UTC m=+50.647952682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5hm57" (UID: "88c77ad8-3d00-4d21-8ac6-d6772afccf31") : configmap "kube-rbac-proxy" not found
Jul 01 19:22:12.419491 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:12.419420    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nrf8m_a0e28d47-acbf-4c6b-ad83-a883ad149185/node-ca/0.log"
Jul 01 19:22:13.213287 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:13.213259    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jjqmq_9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6/serve-healthcheck-canary/0.log"
Jul 01 19:22:15.415869 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:15.415837    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:15.712952 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:15.712913    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:15.820884 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:15.820851    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:15.926475 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:15.926435    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:16.027313 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:16.027231    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:16.128059 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:16.128023    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:16.135991 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:16.135967    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:16.136095 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:16.136012    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:16.136095 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:16.136082    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:22:16.136164 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:16.136103    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:16.136164 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:16.136140    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm podName:88c77ad8-3d00-4d21-8ac6-d6772afccf31 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:24.136124144 +0000 UTC m=+58.682373459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5hm57" (UID: "88c77ad8-3d00-4d21-8ac6-d6772afccf31") : configmap "kube-rbac-proxy" not found
Jul 01 19:22:16.136164 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:16.136153    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls podName:88c77ad8-3d00-4d21-8ac6-d6772afccf31 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:24.136146813 +0000 UTC m=+58.682396126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hm57" (UID: "88c77ad8-3d00-4d21-8ac6-d6772afccf31") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:16.384361 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:16.384269    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:16.739925 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:16.739894    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:16.740384 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:16.740080    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:16.740384 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:16.740101    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-566f556b75-9mp45: secret "image-registry-tls" not found
Jul 01 19:22:16.740384 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:16.740171    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls podName:dafde6e8-47b0-4b6c-aa45-c407774338c5 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:32.740149914 +0000 UTC m=+67.286399227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls") pod "image-registry-566f556b75-9mp45" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5") : secret "image-registry-tls" not found
Jul 01 19:22:16.860561 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:16.860524    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:17.159032 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:17.158953    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:17.547809 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:17.547772    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:17.547961 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:17.547895    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:17.547961 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:17.547955    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls podName:60583eee-9b12-46e0-a65d-d9ca56efc9fb nodeName:}" failed. No retries permitted until 2026-07-01 19:22:33.547940241 +0000 UTC m=+68.094189556 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls") pod "dns-default-sdr2c" (UID: "60583eee-9b12-46e0-a65d-d9ca56efc9fb") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:17.560149 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:17.560120    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:17.756081 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:17.756049    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:18.161922 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:18.161882    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:18.986284 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:18.986249    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:19.325590 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:19.325501    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:19.415324 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:19.415293    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:19.607716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:19.607617    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:19.806907 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:19.806867    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:19.976583 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:19.976551    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:20.049543 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:20.049503    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:20.831170 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:20.831136    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:21.168932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:21.168851    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:21.372764 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:21.372726    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:21.538472 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:21.538436    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:21.608113 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:21.608082    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:21.762811 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:21.762775    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:21.963715 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:21.963671    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:22.361654 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:22.361573    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:22.961415 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:22.961370    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:23.197210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:23.197174    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:23.570428 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:23.570394    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:23.766899 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:23.766871    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:24.163281 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:24.163247    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:24.198761 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:24.198726    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:24.199042 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:24.198837    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:24.199042 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:24.198907    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:24.199042 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:24.198974    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls podName:88c77ad8-3d00-4d21-8ac6-d6772afccf31 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:40.198957984 +0000 UTC m=+74.745207298 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hm57" (UID: "88c77ad8-3d00-4d21-8ac6-d6772afccf31") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:24.199325 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:24.199303    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88c77ad8-3d00-4d21-8ac6-d6772afccf31-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:24.761153 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:24.761120    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:25.198965 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:25.198936    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:25.576457 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:25.576354    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:25.882125 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:25.882031    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:26.168278 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:26.168187    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:26.212619 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:26.212588    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxjj5"
Jul 01 19:22:26.368100 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:26.368059    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:28.607370 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:28.607335    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:28.677582 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:28.677554    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:30.651132 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.651078    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:22:30.651132 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.651136    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:22:30.654836 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.654819    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 19:22:30.654982 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.654966    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 19:22:30.664344 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.664318    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55e6ffec-a820-4802-89cb-747770b45bb9-original-pull-secret\") pod \"global-pull-secret-syncer-5zvjf\" (UID: \"55e6ffec-a820-4802-89cb-747770b45bb9\") " pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:22:30.664344 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.664339    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f41077b2-239b-44f8-be7d-b8b9464b42e9-metrics-certs\") pod \"network-metrics-daemon-jbvd8\" (UID: \"f41077b2-239b-44f8-be7d-b8b9464b42e9\") " pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:22:30.852613 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.852576    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz5s\" (UniqueName: \"kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s\") pod \"network-check-target-bc752\" (UID: \"538fff2a-3657-456b-9158-297890173d5c\") " pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:22:30.855570 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.855553    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 19:22:30.866442 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.866425    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 19:22:30.876943 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.876920    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jz5s\" (UniqueName: \"kubernetes.io/projected/538fff2a-3657-456b-9158-297890173d5c-kube-api-access-6jz5s\") pod \"network-check-target-bc752\" (UID: \"538fff2a-3657-456b-9158-297890173d5c\") " pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:22:30.889539 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.889519    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mzk4c\""
Jul 01 19:22:30.895679 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.895665    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zbvbl\""
Jul 01 19:22:30.897797 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.897769    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbvd8"
Jul 01 19:22:30.897893 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.897781    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zvjf"
Jul 01 19:22:30.903448 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:30.903429    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:22:31.061992 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.061955    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jbvd8"]
Jul 01 19:22:31.066044 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:22:31.066021    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41077b2_239b_44f8_be7d_b8b9464b42e9.slice/crio-12870b4d5de1959dfef1c309a9a4545862c7e9483b2ddcd59b75dc0ac299dad2 WatchSource:0}: Error finding container 12870b4d5de1959dfef1c309a9a4545862c7e9483b2ddcd59b75dc0ac299dad2: Status 404 returned error can't find the container with id 12870b4d5de1959dfef1c309a9a4545862c7e9483b2ddcd59b75dc0ac299dad2
Jul 01 19:22:31.285145 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.285116    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5zvjf"]
Jul 01 19:22:31.287082 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.287064    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bc752"]
Jul 01 19:22:31.288251 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:22:31.288211    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e6ffec_a820_4802_89cb_747770b45bb9.slice/crio-474a54dee76593509f86568a6662a052fbac9dfc984fb3fe547dbec1a659b8a5 WatchSource:0}: Error finding container 474a54dee76593509f86568a6662a052fbac9dfc984fb3fe547dbec1a659b8a5: Status 404 returned error can't find the container with id 474a54dee76593509f86568a6662a052fbac9dfc984fb3fe547dbec1a659b8a5
Jul 01 19:22:31.289123 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:22:31.289105    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538fff2a_3657_456b_9158_297890173d5c.slice/crio-d33886fb7cb1aa79cecbd9d45c357567cb8551e9a1b4ff0c74156e4a33134b40 WatchSource:0}: Error finding container d33886fb7cb1aa79cecbd9d45c357567cb8551e9a1b4ff0c74156e4a33134b40: Status 404 returned error can't find the container with id d33886fb7cb1aa79cecbd9d45c357567cb8551e9a1b4ff0c74156e4a33134b40
Jul 01 19:22:31.307508 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.307482    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bc752" event={"ID":"538fff2a-3657-456b-9158-297890173d5c","Type":"ContainerStarted","Data":"d33886fb7cb1aa79cecbd9d45c357567cb8551e9a1b4ff0c74156e4a33134b40"}
Jul 01 19:22:31.308539 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.308516    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5zvjf" event={"ID":"55e6ffec-a820-4802-89cb-747770b45bb9","Type":"ContainerStarted","Data":"474a54dee76593509f86568a6662a052fbac9dfc984fb3fe547dbec1a659b8a5"}
Jul 01 19:22:31.309488 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.309468    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbvd8" event={"ID":"f41077b2-239b-44f8-be7d-b8b9464b42e9","Type":"ContainerStarted","Data":"12870b4d5de1959dfef1c309a9a4545862c7e9483b2ddcd59b75dc0ac299dad2"}
Jul 01 19:22:31.676347 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.676269    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"]
Jul 01 19:22:31.707786 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.707760    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:31.721467 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.721446    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 19:22:31.722500 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.722483    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 01 19:22:31.722579 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.722496    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 19:22:31.722579 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.722496    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 19:22:31.735040 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.735015    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"]
Jul 01 19:22:31.740650 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.740589    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"]
Jul 01 19:22:31.756635 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.756611    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-566f556b75-9mp45"]
Jul 01 19:22:31.756768 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.756662    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.756901 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:31.756859    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-566f556b75-9mp45" podUID="dafde6e8-47b0-4b6c-aa45-c407774338c5"
Jul 01 19:22:31.759031 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.759007    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98mn\" (UniqueName: \"kubernetes.io/projected/e5887a73-9717-45be-9229-000b08b8ebf9-kube-api-access-n98mn\") pod \"klusterlet-addon-workmgr-77f759b9d5-56bst\" (UID: \"e5887a73-9717-45be-9229-000b08b8ebf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:31.759147 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.759073    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e5887a73-9717-45be-9229-000b08b8ebf9-klusterlet-config\") pod \"klusterlet-addon-workmgr-77f759b9d5-56bst\" (UID: \"e5887a73-9717-45be-9229-000b08b8ebf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:31.759147 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.759105    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5887a73-9717-45be-9229-000b08b8ebf9-tmp\") pod \"klusterlet-addon-workmgr-77f759b9d5-56bst\" (UID: \"e5887a73-9717-45be-9229-000b08b8ebf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:31.775635 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.775613    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 01 19:22:31.775797 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.775782    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 01 19:22:31.776089 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.776068    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 01 19:22:31.780585 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.780566    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 01 19:22:31.809205 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.809174    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"]
Jul 01 19:22:31.860089 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.859558    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98mn\" (UniqueName: \"kubernetes.io/projected/e5887a73-9717-45be-9229-000b08b8ebf9-kube-api-access-n98mn\") pod \"klusterlet-addon-workmgr-77f759b9d5-56bst\" (UID: \"e5887a73-9717-45be-9229-000b08b8ebf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:31.860089 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.859605    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5887a73-9717-45be-9229-000b08b8ebf9-tmp\") pod \"klusterlet-addon-workmgr-77f759b9d5-56bst\" (UID: \"e5887a73-9717-45be-9229-000b08b8ebf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:31.860089 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.859639    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.860089 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.859677    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e5887a73-9717-45be-9229-000b08b8ebf9-klusterlet-config\") pod \"klusterlet-addon-workmgr-77f759b9d5-56bst\" (UID: \"e5887a73-9717-45be-9229-000b08b8ebf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:31.860089 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.859717    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-hub\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.860089 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.859765    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-ca\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.860089 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.859787    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/720c5945-9bee-402a-940d-803f440d9fda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.860089 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.859825    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.860089 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.859873    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd5ml\" (UniqueName: \"kubernetes.io/projected/720c5945-9bee-402a-940d-803f440d9fda-kube-api-access-vd5ml\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.860629 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.860274    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5887a73-9717-45be-9229-000b08b8ebf9-tmp\") pod \"klusterlet-addon-workmgr-77f759b9d5-56bst\" (UID: \"e5887a73-9717-45be-9229-000b08b8ebf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:31.862983 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.862952    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e5887a73-9717-45be-9229-000b08b8ebf9-klusterlet-config\") pod \"klusterlet-addon-workmgr-77f759b9d5-56bst\" (UID: \"e5887a73-9717-45be-9229-000b08b8ebf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:31.899045 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.898989    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98mn\" (UniqueName: \"kubernetes.io/projected/e5887a73-9717-45be-9229-000b08b8ebf9-kube-api-access-n98mn\") pod \"klusterlet-addon-workmgr-77f759b9d5-56bst\" (UID: \"e5887a73-9717-45be-9229-000b08b8ebf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:31.942757 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.942706    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:31.960950 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.960921    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.961103 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.960976    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-hub\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.961103 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.961033    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-ca\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.961103 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.961061    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/720c5945-9bee-402a-940d-803f440d9fda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.961270 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.961105    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.961270 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.961194    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd5ml\" (UniqueName: \"kubernetes.io/projected/720c5945-9bee-402a-940d-803f440d9fda-kube-api-access-vd5ml\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.962183 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.962154    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/720c5945-9bee-402a-940d-803f440d9fda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.965073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.964561    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-ca\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.965073 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.965029    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.965247 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.965217    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.966745 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.966611    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/720c5945-9bee-402a-940d-803f440d9fda-hub\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:31.973881 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:31.973853    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd5ml\" (UniqueName: \"kubernetes.io/projected/720c5945-9bee-402a-940d-803f440d9fda-kube-api-access-vd5ml\") pod \"cluster-proxy-proxy-agent-679f6785d6-9h2kj\" (UID: \"720c5945-9bee-402a-940d-803f440d9fda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:32.016125 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.016089    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:32.076313 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.076108    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"
Jul 01 19:22:32.115355 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.115302    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:32.208449 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.208355    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"]
Jul 01 19:22:32.264295 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.264210    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:32.313159 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.313130    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:32.355529 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.355218    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:32.398450 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.398353    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:32.438811 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:22:32.438768    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5887a73_9717_45be_9229_000b08b8ebf9.slice/crio-428fd33e5eec952461e47b893489389da72373b8f27123556ad3b2a1c8b4b2b8 WatchSource:0}: Error finding container 428fd33e5eec952461e47b893489389da72373b8f27123556ad3b2a1c8b4b2b8: Status 404 returned error can't find the container with id 428fd33e5eec952461e47b893489389da72373b8f27123556ad3b2a1c8b4b2b8
Jul 01 19:22:32.465255 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.465180    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-certificates\") pod \"dafde6e8-47b0-4b6c-aa45-c407774338c5\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") "
Jul 01 19:22:32.465255 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.465218    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-installation-pull-secrets\") pod \"dafde6e8-47b0-4b6c-aa45-c407774338c5\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") "
Jul 01 19:22:32.465255 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.465251    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffkn8\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-kube-api-access-ffkn8\") pod \"dafde6e8-47b0-4b6c-aa45-c407774338c5\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") "
Jul 01 19:22:32.465516 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.465274    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-image-registry-private-configuration\") pod \"dafde6e8-47b0-4b6c-aa45-c407774338c5\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") "
Jul 01 19:22:32.465516 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.465293    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dafde6e8-47b0-4b6c-aa45-c407774338c5-ca-trust-extracted\") pod \"dafde6e8-47b0-4b6c-aa45-c407774338c5\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") "
Jul 01 19:22:32.465516 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.465338    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-bound-sa-token\") pod \"dafde6e8-47b0-4b6c-aa45-c407774338c5\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") "
Jul 01 19:22:32.465516 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.465371    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-trusted-ca\") pod \"dafde6e8-47b0-4b6c-aa45-c407774338c5\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") "
Jul 01 19:22:32.465742 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.465601    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dafde6e8-47b0-4b6c-aa45-c407774338c5" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:22:32.465742 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.465710    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafde6e8-47b0-4b6c-aa45-c407774338c5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dafde6e8-47b0-4b6c-aa45-c407774338c5" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:22:32.466559 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.466530    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dafde6e8-47b0-4b6c-aa45-c407774338c5" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:22:32.468147 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.468119    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dafde6e8-47b0-4b6c-aa45-c407774338c5" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:22:32.468279 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.468244    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-kube-api-access-ffkn8" (OuterVolumeSpecName: "kube-api-access-ffkn8") pod "dafde6e8-47b0-4b6c-aa45-c407774338c5" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5"). InnerVolumeSpecName "kube-api-access-ffkn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:22:32.468343 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.468294    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dafde6e8-47b0-4b6c-aa45-c407774338c5" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:22:32.468464 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.468432    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dafde6e8-47b0-4b6c-aa45-c407774338c5" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:22:32.565993 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.565964    2586 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-bound-sa-token\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:22:32.565993 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.565994    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-trusted-ca\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:22:32.566210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.566008    2586 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-certificates\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:22:32.566210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.566022    2586 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-installation-pull-secrets\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:22:32.566210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.566037    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffkn8\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-kube-api-access-ffkn8\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:22:32.566210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.566050    2586 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dafde6e8-47b0-4b6c-aa45-c407774338c5-image-registry-private-configuration\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:22:32.566210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.566065    2586 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dafde6e8-47b0-4b6c-aa45-c407774338c5-ca-trust-extracted\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:22:32.592239 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.592199    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj"]
Jul 01 19:22:32.747153 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:22:32.747077    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod720c5945_9bee_402a_940d_803f440d9fda.slice/crio-98ecfd3a0b40d0853a0c7a6445b31b0778b76f9062abe16af521ff7927426cb7 WatchSource:0}: Error finding container 98ecfd3a0b40d0853a0c7a6445b31b0778b76f9062abe16af521ff7927426cb7: Status 404 returned error can't find the container with id 98ecfd3a0b40d0853a0c7a6445b31b0778b76f9062abe16af521ff7927426cb7
Jul 01 19:22:32.768035 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.768000    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:32.772914 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.772857    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls\") pod \"image-registry-566f556b75-9mp45\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") " pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:32.868711 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.868667    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls\") pod \"dafde6e8-47b0-4b6c-aa45-c407774338c5\" (UID: \"dafde6e8-47b0-4b6c-aa45-c407774338c5\") "
Jul 01 19:22:32.871119 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.871090    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dafde6e8-47b0-4b6c-aa45-c407774338c5" (UID: "dafde6e8-47b0-4b6c-aa45-c407774338c5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:22:32.969390 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:32.969357    2586 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dafde6e8-47b0-4b6c-aa45-c407774338c5-registry-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:22:33.321734 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.321150    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbvd8" event={"ID":"f41077b2-239b-44f8-be7d-b8b9464b42e9","Type":"ContainerStarted","Data":"b43cdf1362273229aac88c86d652eccb82cf66e8f046d0008af69875808dd3ab"}
Jul 01 19:22:33.321734 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.321214    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbvd8" event={"ID":"f41077b2-239b-44f8-be7d-b8b9464b42e9","Type":"ContainerStarted","Data":"18204241c24be66646b50fe8e3f477ef40b53554e596bbcae7862cce4ff2c607"}
Jul 01 19:22:33.323054 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.322940    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst" event={"ID":"e5887a73-9717-45be-9229-000b08b8ebf9","Type":"ContainerStarted","Data":"428fd33e5eec952461e47b893489389da72373b8f27123556ad3b2a1c8b4b2b8"}
Jul 01 19:22:33.327008 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.325486    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566f556b75-9mp45"
Jul 01 19:22:33.327008 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.326600    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj" event={"ID":"720c5945-9bee-402a-940d-803f440d9fda","Type":"ContainerStarted","Data":"98ecfd3a0b40d0853a0c7a6445b31b0778b76f9062abe16af521ff7927426cb7"}
Jul 01 19:22:33.360724 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.360127    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jbvd8" podStartSLOduration=65.651550457 podStartE2EDuration="1m7.360103832s" totalImagesPullingTime="1.708553375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:33.359252828 +0000 UTC m=+67.905502155" watchObservedRunningTime="2026-07-01 19:22:33.360103832 +0000 UTC m=+67.906353167"
Jul 01 19:22:33.413634 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.413568    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-566f556b75-9mp45"]
Jul 01 19:22:33.419557 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.419502    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-566f556b75-9mp45"]
Jul 01 19:22:33.508000 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.507964    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:33.575399 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.574825    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:33.578930 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.578833    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60583eee-9b12-46e0-a65d-d9ca56efc9fb-metrics-tls\") pod \"dns-default-sdr2c\" (UID: \"60583eee-9b12-46e0-a65d-d9ca56efc9fb\") " pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:33.605938 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.605876    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:33.868996 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.868919    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f8prn\""
Jul 01 19:22:33.877716 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.877440    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:33.885724 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:33.885501    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dafde6e8-47b0-4b6c-aa45-c407774338c5" path="/var/lib/kubelet/pods/dafde6e8-47b0-4b6c-aa45-c407774338c5/volumes"
Jul 01 19:22:35.464348 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:35.464308    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:35.575016 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:35.574933    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:35.713737 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:35.713675    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:35.881531 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:35.881458    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:36.981794 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:36.981746    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:37.081830 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:37.081782    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:38.584355 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:38.584313    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:38.645261 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:38.645223    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:40.228881 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:40.228853    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:40.231008 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:40.230989    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88c77ad8-3d00-4d21-8ac6-d6772afccf31-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hm57\" (UID: \"88c77ad8-3d00-4d21-8ac6-d6772afccf31\") " pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:40.260894 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:40.260870    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tz26f\""
Jul 01 19:22:40.269186 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:40.269165    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5hm57"
Jul 01 19:22:40.674945 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:40.674919    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sdr2c"]
Jul 01 19:22:40.682788 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:22:40.682760    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60583eee_9b12_46e0_a65d_d9ca56efc9fb.slice/crio-6962509a4e077bdb45e22841e1060b09c1f2ad6c1a8c18526f7a0b29aababe8e WatchSource:0}: Error finding container 6962509a4e077bdb45e22841e1060b09c1f2ad6c1a8c18526f7a0b29aababe8e: Status 404 returned error can't find the container with id 6962509a4e077bdb45e22841e1060b09c1f2ad6c1a8c18526f7a0b29aababe8e
Jul 01 19:22:40.697885 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:40.697853    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5hm57"]
Jul 01 19:22:41.353815 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.353780    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj" event={"ID":"720c5945-9bee-402a-940d-803f440d9fda","Type":"ContainerStarted","Data":"b4c818598ef9dc378d7a4b360c5d457578947cad16f50b0b2be7257e815ba8cf"}
Jul 01 19:22:41.354222 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.353829    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj" event={"ID":"720c5945-9bee-402a-940d-803f440d9fda","Type":"ContainerStarted","Data":"d772be3318d33ff6af9e049ba07807955fdf9ab79b9ddacde86b9f2e8f4a56a8"}
Jul 01 19:22:41.354222 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.353851    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj" event={"ID":"720c5945-9bee-402a-940d-803f440d9fda","Type":"ContainerStarted","Data":"8c044c34384b0a338120d03b69c0970bfd614763467b32d2986a2364df76a7c6"}
Jul 01 19:22:41.354964 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.354935    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sdr2c" event={"ID":"60583eee-9b12-46e0-a65d-d9ca56efc9fb","Type":"ContainerStarted","Data":"6962509a4e077bdb45e22841e1060b09c1f2ad6c1a8c18526f7a0b29aababe8e"}
Jul 01 19:22:41.356271 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.356249    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bc752" event={"ID":"538fff2a-3657-456b-9158-297890173d5c","Type":"ContainerStarted","Data":"ec53c0fcfbd131ab4aee9fde7620b87ce61c3c09bd94438d154d4be389e36578"}
Jul 01 19:22:41.356371 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.356289    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:22:41.357762 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.357740    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5zvjf" event={"ID":"55e6ffec-a820-4802-89cb-747770b45bb9","Type":"ContainerStarted","Data":"3a6c182d01cd679c51e220e3e2d006f514903f15aba67f14f971d1c5dbbf0590"}
Jul 01 19:22:41.359104 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.359081    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst" event={"ID":"e5887a73-9717-45be-9229-000b08b8ebf9","Type":"ContainerStarted","Data":"544461b573bce68f4b2a5071d74904109648703e06a951b6ce9a4eea1de92a19"}
Jul 01 19:22:41.359225 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.359210    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:41.360236 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.360218    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5hm57" event={"ID":"88c77ad8-3d00-4d21-8ac6-d6772afccf31","Type":"ContainerStarted","Data":"2e2ffd61aa80fd7729e3aef76a2155c47898ed417015069da9d013649d5896b6"}
Jul 01 19:22:41.361068 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.361052    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst"
Jul 01 19:22:41.380053 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.379999    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-679f6785d6-9h2kj" podStartSLOduration=2.59831757 podStartE2EDuration="10.379979969s" totalImagesPullingTime="7.781662399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:41.379476956 +0000 UTC m=+75.925726293" watchObservedRunningTime="2026-07-01 19:22:41.379979969 +0000 UTC m=+75.926229304"
Jul 01 19:22:41.427214 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.427159    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5zvjf" podStartSLOduration=66.182244755 podStartE2EDuration="1m15.42713788s" totalImagesPullingTime="9.244893125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:41.398089068 +0000 UTC m=+75.944338405" watchObservedRunningTime="2026-07-01 19:22:41.42713788 +0000 UTC m=+75.973387217"
Jul 01 19:22:41.427379 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.427309    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bc752" podStartSLOduration=66.183189579 podStartE2EDuration="1m15.427302597s" totalImagesPullingTime="9.244113018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:41.425453502 +0000 UTC m=+75.971702843" watchObservedRunningTime="2026-07-01 19:22:41.427302597 +0000 UTC m=+75.973551933"
Jul 01 19:22:41.449905 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.449849    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77f759b9d5-56bst" podStartSLOduration=2.356123092 podStartE2EDuration="10.449833083s" totalImagesPullingTime="8.093709991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:41.448361409 +0000 UTC m=+75.994610747" watchObservedRunningTime="2026-07-01 19:22:41.449833083 +0000 UTC m=+75.996082414"
Jul 01 19:22:41.766113 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.766009    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:41.863949 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.863913    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:41.945635 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:41.945599    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:42.620402 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:42.620364    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:42.768250 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:42.768086    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:43.371475 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:43.371441    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sdr2c" event={"ID":"60583eee-9b12-46e0-a65d-d9ca56efc9fb","Type":"ContainerStarted","Data":"2ef0009c98c96087d9fdd10429d729b85cc5e45f2fadf0617def7dc93a992924"}
Jul 01 19:22:43.371649 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:43.371501    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sdr2c" event={"ID":"60583eee-9b12-46e0-a65d-d9ca56efc9fb","Type":"ContainerStarted","Data":"0d6c155c1c95ae645d571f75b0a6367e7c01f0c9e688592310b89002afac8ab2"}
Jul 01 19:22:43.371649 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:43.371603    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:43.394715 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:43.394644    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sdr2c" podStartSLOduration=41.350086058 podStartE2EDuration="43.394622778s" totalImagesPullingTime="2.04453672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:43.393401542 +0000 UTC m=+77.939650929" watchObservedRunningTime="2026-07-01 19:22:43.394622778 +0000 UTC m=+77.940872115"
Jul 01 19:22:43.404833 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:43.404803    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:43.517589 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:43.517553    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:45.382288 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:45.382208    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5hm57" event={"ID":"88c77ad8-3d00-4d21-8ac6-d6772afccf31","Type":"ContainerStarted","Data":"08bd20764ecbb489ff8f61fa369a52dd68a0fa1978f305498b8c7fc7baa18bd0"}
Jul 01 19:22:45.580598 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:45.580554    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:46.388841 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:46.388805    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5hm57" event={"ID":"88c77ad8-3d00-4d21-8ac6-d6772afccf31","Type":"ContainerStarted","Data":"0b0f8eacdb89857e28b3b31c931a2eb6c685515d73087dabd71163ad1150e010"}
Jul 01 19:22:47.659579 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:47.659458    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:47.746840 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:47.746806    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:47.943060 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:47.943019    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:48.167533 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.167503    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:48.216968 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.216889    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-8kssh"]
Jul 01 19:22:48.227557 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.227533    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.230392 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.230333    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 01 19:22:48.231625 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.231580    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jtxwm\""
Jul 01 19:22:48.232654 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.232191    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 19:22:48.232654 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.232492    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 01 19:22:48.232654 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.232512    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 01 19:22:48.233453 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.233142    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 19:22:48.233453 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.233287    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 19:22:48.235987 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.235962    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-8kssh"]
Jul 01 19:22:48.237235 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.237198    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8gks2"]
Jul 01 19:22:48.240871 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.240851    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.243435 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.243413    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 19:22:48.243665 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.243632    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 19:22:48.243843 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.243819    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 19:22:48.243918 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.243903    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k8ffm\""
Jul 01 19:22:48.273514 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.273483    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:48.394102 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394070    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-textfile\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.394254 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394130    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.394254 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394156    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkt5h\" (UniqueName: \"kubernetes.io/projected/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-api-access-wkt5h\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.394254 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394187    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-metrics-client-ca\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.394254 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394211    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1e76d90-156f-4f30-95c7-bfcc96ee0784-root\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.394254 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394240    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-accelerators-collector-config\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.394472 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394288    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4vs6\" (UniqueName: \"kubernetes.io/projected/a1e76d90-156f-4f30-95c7-bfcc96ee0784-kube-api-access-s4vs6\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.394472 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394310    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e76d90-156f-4f30-95c7-bfcc96ee0784-sys\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.394472 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394330    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1e76d90-156f-4f30-95c7-bfcc96ee0784-metrics-client-ca\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.394472 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394346    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-tls\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.394472 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394366    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.394472 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394462    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.394757 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394498    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.394757 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394542    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-wtmp\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.394757 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.394574    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.397740 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.397716    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5hm57" event={"ID":"88c77ad8-3d00-4d21-8ac6-d6772afccf31","Type":"ContainerStarted","Data":"2e3dcb230afe4d3d602a9161943819e83b1822a727c70fc9717d834a906cf082"}
Jul 01 19:22:48.415888 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.415840    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5hm57" podStartSLOduration=32.920128093 podStartE2EDuration="40.415822749s" totalImagesPullingTime="7.495694656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:48.415361317 +0000 UTC m=+82.961610655" watchObservedRunningTime="2026-07-01 19:22:48.415822749 +0000 UTC m=+82.962072085"
Jul 01 19:22:48.495615 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.495526    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-wtmp\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.495615 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.495567    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.495874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.495716    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-wtmp\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.495874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.495739    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-textfile\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.495874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.495774    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.495874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.495798    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkt5h\" (UniqueName: \"kubernetes.io/projected/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-api-access-wkt5h\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.495874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.495826    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-metrics-client-ca\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.495874 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.495862    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1e76d90-156f-4f30-95c7-bfcc96ee0784-root\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.496178 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.495912    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-accelerators-collector-config\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.496178 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.495950    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4vs6\" (UniqueName: \"kubernetes.io/projected/a1e76d90-156f-4f30-95c7-bfcc96ee0784-kube-api-access-s4vs6\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.496178 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.495991    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e76d90-156f-4f30-95c7-bfcc96ee0784-sys\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.496178 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.496019    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1e76d90-156f-4f30-95c7-bfcc96ee0784-metrics-client-ca\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.496178 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.496045    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-tls\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.496178 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.496077    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.496178 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.496114    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.496178 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.496140    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.496178 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.496168    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-textfile\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.496650 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.496186    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.496650 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.496195    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e76d90-156f-4f30-95c7-bfcc96ee0784-sys\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.496650 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:48.496254    2586 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 01 19:22:48.496650 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:48.496275    2586 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 01 19:22:48.496650 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:48.496316    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-tls podName:a1e76d90-156f-4f30-95c7-bfcc96ee0784 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:48.996297046 +0000 UTC m=+83.542546372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-tls") pod "node-exporter-8gks2" (UID: "a1e76d90-156f-4f30-95c7-bfcc96ee0784") : secret "node-exporter-tls" not found
Jul 01 19:22:48.496650 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:22:48.496370    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-tls podName:849b87f9-bd2e-41c4-acd7-23fbcea88f8d nodeName:}" failed. No retries permitted until 2026-07-01 19:22:48.996353506 +0000 UTC m=+83.542602839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-tls") pod "kube-state-metrics-845b954455-8kssh" (UID: "849b87f9-bd2e-41c4-acd7-23fbcea88f8d") : secret "kube-state-metrics-tls" not found
Jul 01 19:22:48.496650 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.496617    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1e76d90-156f-4f30-95c7-bfcc96ee0784-metrics-client-ca\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.496650 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.496637    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1e76d90-156f-4f30-95c7-bfcc96ee0784-root\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.497122 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.496934    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-accelerators-collector-config\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.497122 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.497079    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.497234 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.497203    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-metrics-client-ca\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.498637 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.498615    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.498783 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.498615    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.528843 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.528815    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkt5h\" (UniqueName: \"kubernetes.io/projected/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-api-access-wkt5h\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:48.528987 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.528852    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4vs6\" (UniqueName: \"kubernetes.io/projected/a1e76d90-156f-4f30-95c7-bfcc96ee0784-kube-api-access-s4vs6\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:48.600964 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.600928    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:48.701070 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.701031    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:48.819397 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.819321    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:48.952829 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:48.952801    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:49.000299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:49.000267    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-tls\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:49.000299 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:49.000302    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:49.002725 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:49.002677    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1e76d90-156f-4f30-95c7-bfcc96ee0784-node-exporter-tls\") pod \"node-exporter-8gks2\" (UID: \"a1e76d90-156f-4f30-95c7-bfcc96ee0784\") " pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:49.002833 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:49.002741    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/849b87f9-bd2e-41c4-acd7-23fbcea88f8d-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-8kssh\" (UID: \"849b87f9-bd2e-41c4-acd7-23fbcea88f8d\") " pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:49.143618 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:49.143530    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh"
Jul 01 19:22:49.152655 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:49.152631    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8gks2"
Jul 01 19:22:49.162181 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:22:49.162150    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e76d90_156f_4f30_95c7_bfcc96ee0784.slice/crio-9ac8704ed0313a828f7c231f771187e283047807616aa99adb344181b2a0d696 WatchSource:0}: Error finding container 9ac8704ed0313a828f7c231f771187e283047807616aa99adb344181b2a0d696: Status 404 returned error can't find the container with id 9ac8704ed0313a828f7c231f771187e283047807616aa99adb344181b2a0d696
Jul 01 19:22:49.372972 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:22:49.372942    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849b87f9_bd2e_41c4_acd7_23fbcea88f8d.slice/crio-3bc2c7576afcfc0d113965376bdbac569452fb533463ad3f89af770f990e4acb WatchSource:0}: Error finding container 3bc2c7576afcfc0d113965376bdbac569452fb533463ad3f89af770f990e4acb: Status 404 returned error can't find the container with id 3bc2c7576afcfc0d113965376bdbac569452fb533463ad3f89af770f990e4acb
Jul 01 19:22:49.380696 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:49.380660    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-8kssh"]
Jul 01 19:22:49.402336 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:49.402261    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8gks2" event={"ID":"a1e76d90-156f-4f30-95c7-bfcc96ee0784","Type":"ContainerStarted","Data":"9ac8704ed0313a828f7c231f771187e283047807616aa99adb344181b2a0d696"}
Jul 01 19:22:49.403419 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:49.403395    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh" event={"ID":"849b87f9-bd2e-41c4-acd7-23fbcea88f8d","Type":"ContainerStarted","Data":"3bc2c7576afcfc0d113965376bdbac569452fb533463ad3f89af770f990e4acb"}
Jul 01 19:22:49.682016 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:49.681932    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:49.971372 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:49.971339    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:50.151743 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:50.151713    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:50.409635 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:50.409554    2586 generic.go:356] "Generic (PLEG): container finished" podID="a1e76d90-156f-4f30-95c7-bfcc96ee0784" containerID="be7d5fd2c28cc6399a6b2a15a803e4568d86f77803988ee00a3dc5ce0f63b2f9" exitCode=0
Jul 01 19:22:50.409813 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:50.409643    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8gks2" event={"ID":"a1e76d90-156f-4f30-95c7-bfcc96ee0784","Type":"ContainerDied","Data":"be7d5fd2c28cc6399a6b2a15a803e4568d86f77803988ee00a3dc5ce0f63b2f9"}
Jul 01 19:22:51.414755 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:51.414644    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh" event={"ID":"849b87f9-bd2e-41c4-acd7-23fbcea88f8d","Type":"ContainerStarted","Data":"cf888e84a92e0b71b44c3f6d9154992c0fc99d5cf2145df4486a618d1185f95e"}
Jul 01 19:22:51.415181 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:51.414766    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh" event={"ID":"849b87f9-bd2e-41c4-acd7-23fbcea88f8d","Type":"ContainerStarted","Data":"c7fe2ffe1d8ff59a7fca38e31adf51b71bf3313b85cad05df44f7989cf125e8d"}
Jul 01 19:22:51.415181 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:51.414798    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh" event={"ID":"849b87f9-bd2e-41c4-acd7-23fbcea88f8d","Type":"ContainerStarted","Data":"a3018fbacad63243dde752100f1bc9d2bb2b6a8603b432e7e0b5d948b2c36aea"}
Jul 01 19:22:51.417060 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:51.417035    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8gks2" event={"ID":"a1e76d90-156f-4f30-95c7-bfcc96ee0784","Type":"ContainerStarted","Data":"ce0d806fde4a49595715dab760269063027ed2a4055b92ca1965e27d5c281328"}
Jul 01 19:22:51.417166 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:51.417076    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8gks2" event={"ID":"a1e76d90-156f-4f30-95c7-bfcc96ee0784","Type":"ContainerStarted","Data":"9814a10ab2133307e5ac7844d50d5a1956e97884c6d5417a63675c5d599025b4"}
Jul 01 19:22:51.467707 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:51.467645    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-8kssh" podStartSLOduration=1.784885897 podStartE2EDuration="3.467628328s" totalImagesPullingTime="1.682742431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:51.467561855 +0000 UTC m=+86.013811191" watchObservedRunningTime="2026-07-01 19:22:51.467628328 +0000 UTC m=+86.013877665"
Jul 01 19:22:51.532522 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:51.532441    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8gks2" podStartSLOduration=2.656305719 podStartE2EDuration="3.532421231s" totalImagesPullingTime="810.084ms" totalInitContainerRuntime="66.031512ms" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:51.531517111 +0000 UTC m=+86.077766447" watchObservedRunningTime="2026-07-01 19:22:51.532421231 +0000 UTC m=+86.078670570"
Jul 01 19:22:52.022056 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:52.022026    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:52.412059 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:52.411960    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:52.687915 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:52.687819    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:52.924776 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:52.924742    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:53.200904 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:53.200854    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:53.377753 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:53.377725    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sdr2c"
Jul 01 19:22:53.578410 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:53.578328    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:54.156303 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:54.156271    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:54.375452 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:54.375410    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:54.553928 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:54.553894    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:54.754390 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:54.754352    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:54.965099 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:54.965068    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:55.368994 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:55.368910    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:55.647933 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:55.647849    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:55.857440 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:55.857411    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.031301 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:57.031264    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.191116 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:57.191084    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.304250 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:57.304171    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.407795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:57.407755    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.643368 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:57.643284    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.802479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:57.802441    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.894524 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:57.894441    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:58.418546 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:58.418517    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:58.673500 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:58.673410    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:58.848130 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:58.848094    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:58.919365 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:58.919325    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:59.057976 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:59.057945    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:59.216194 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:59.216150    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:59.317140 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:22:59.317037    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:02.649925 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:02.649895    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:02.722047 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:02.722009    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:03.030460 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.030433    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b5c49974-9j957"]
Jul 01 19:23:03.035811 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.035792    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.038287 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.038265    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 19:23:03.038408 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.038328    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 01 19:23:03.039189 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.039154    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 01 19:23:03.039298 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.039157    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 01 19:23:03.039298 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.039215    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jnpqh\""
Jul 01 19:23:03.039298 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.039279    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 01 19:23:03.039421 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.039302    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 01 19:23:03.039421 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.039402    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 19:23:03.044150 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.044132    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5c49974-9j957"]
Jul 01 19:23:03.172384 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.172341    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:03.211900 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.211877    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgx2x\" (UniqueName: \"kubernetes.io/projected/5565d4e3-9657-4dcb-854c-cdfe497459bc-kube-api-access-xgx2x\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.212046 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.211916    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-config\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.212046 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.211937    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-service-ca\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.212046 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.212015    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-oauth-config\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.212154 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.212078    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-oauth-serving-cert\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.212154 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.212108    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-serving-cert\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.248290 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.248256    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:03.312605 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.312536    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-config\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.312605 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.312571    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-service-ca\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.312605 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.312597    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-oauth-config\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.312871 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.312617    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-oauth-serving-cert\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.312871 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.312639    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-serving-cert\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.312871 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.312675    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgx2x\" (UniqueName: \"kubernetes.io/projected/5565d4e3-9657-4dcb-854c-cdfe497459bc-kube-api-access-xgx2x\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.313384 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.313363    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-oauth-serving-cert\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.313384 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.313390    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-config\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.313567 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.313365    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-service-ca\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.315029 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.315000    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-serving-cert\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.315029 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.315013    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-oauth-config\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.323901 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.323877    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgx2x\" (UniqueName: \"kubernetes.io/projected/5565d4e3-9657-4dcb-854c-cdfe497459bc-kube-api-access-xgx2x\") pod \"console-b5c49974-9j957\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") " pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.346581 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.346559    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:03.475171 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:03.475142    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5c49974-9j957"]
Jul 01 19:23:03.478102 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:23:03.478079    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5565d4e3_9657_4dcb_854c_cdfe497459bc.slice/crio-22f87442148074e6497431764392b26f3057fdcbbe95623c4a227b90542ed82d WatchSource:0}: Error finding container 22f87442148074e6497431764392b26f3057fdcbbe95623c4a227b90542ed82d: Status 404 returned error can't find the container with id 22f87442148074e6497431764392b26f3057fdcbbe95623c4a227b90542ed82d
Jul 01 19:23:04.463295 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:04.463253    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5c49974-9j957" event={"ID":"5565d4e3-9657-4dcb-854c-cdfe497459bc","Type":"ContainerStarted","Data":"22f87442148074e6497431764392b26f3057fdcbbe95623c4a227b90542ed82d"}
Jul 01 19:23:07.476517 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:07.476474    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5c49974-9j957" event={"ID":"5565d4e3-9657-4dcb-854c-cdfe497459bc","Type":"ContainerStarted","Data":"3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb"}
Jul 01 19:23:07.495754 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:07.495700    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b5c49974-9j957" podStartSLOduration=1.24830552 podStartE2EDuration="4.495668364s" totalImagesPullingTime="3.247362844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:07.494034919 +0000 UTC m=+102.040284256" watchObservedRunningTime="2026-07-01 19:23:07.495668364 +0000 UTC m=+102.041917699"
Jul 01 19:23:11.623499 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:11.623460    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:11.700263 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:11.700229    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:12.369092 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:12.369056    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bc752"
Jul 01 19:23:13.346969 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:13.346934    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:13.347435 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:13.347005    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:13.351629 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:13.351603    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:13.498922 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:13.498894    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:13.587557 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:13.587513    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:13.682435 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:13.682338    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:15.683515 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:15.683483    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:22.423018 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:22.422977    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:22.491560 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:22.491520    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:23.538350 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:23.538319    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5c49974-9j957"]
Jul 01 19:23:24.422026 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:24.421984    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:25.424875 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:25.424843    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:26.240034 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:26.239997    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:26.299900 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:26.299866    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:26.646619 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:26.646533    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:26.716394 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:26.716359    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:28.662894 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:28.662859    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:28.731015 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:28.730973    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:48.469755 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.469717    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:48.558538 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.558478    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-b5c49974-9j957" podUID="5565d4e3-9657-4dcb-854c-cdfe497459bc" containerName="console" containerID="cri-o://3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb" gracePeriod=15
Jul 01 19:23:48.791188 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.791165    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:48.832508 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.832479    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-serving-cert\") pod \"5565d4e3-9657-4dcb-854c-cdfe497459bc\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") "
Jul 01 19:23:48.832641 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.832527    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgx2x\" (UniqueName: \"kubernetes.io/projected/5565d4e3-9657-4dcb-854c-cdfe497459bc-kube-api-access-xgx2x\") pod \"5565d4e3-9657-4dcb-854c-cdfe497459bc\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") "
Jul 01 19:23:48.832641 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.832556    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-config\") pod \"5565d4e3-9657-4dcb-854c-cdfe497459bc\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") "
Jul 01 19:23:48.832788 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.832640    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-oauth-serving-cert\") pod \"5565d4e3-9657-4dcb-854c-cdfe497459bc\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") "
Jul 01 19:23:48.832788 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.832724    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-service-ca\") pod \"5565d4e3-9657-4dcb-854c-cdfe497459bc\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") "
Jul 01 19:23:48.832788 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.832757    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-oauth-config\") pod \"5565d4e3-9657-4dcb-854c-cdfe497459bc\" (UID: \"5565d4e3-9657-4dcb-854c-cdfe497459bc\") "
Jul 01 19:23:48.833018 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.832992    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-config" (OuterVolumeSpecName: "console-config") pod "5565d4e3-9657-4dcb-854c-cdfe497459bc" (UID: "5565d4e3-9657-4dcb-854c-cdfe497459bc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:23:48.833139 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.833108    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-service-ca" (OuterVolumeSpecName: "service-ca") pod "5565d4e3-9657-4dcb-854c-cdfe497459bc" (UID: "5565d4e3-9657-4dcb-854c-cdfe497459bc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:23:48.833201 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.833110    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5565d4e3-9657-4dcb-854c-cdfe497459bc" (UID: "5565d4e3-9657-4dcb-854c-cdfe497459bc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:23:48.834777 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.834749    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5565d4e3-9657-4dcb-854c-cdfe497459bc-kube-api-access-xgx2x" (OuterVolumeSpecName: "kube-api-access-xgx2x") pod "5565d4e3-9657-4dcb-854c-cdfe497459bc" (UID: "5565d4e3-9657-4dcb-854c-cdfe497459bc"). InnerVolumeSpecName "kube-api-access-xgx2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:23:48.834857 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.834771    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5565d4e3-9657-4dcb-854c-cdfe497459bc" (UID: "5565d4e3-9657-4dcb-854c-cdfe497459bc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:48.834857 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.834801    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5565d4e3-9657-4dcb-854c-cdfe497459bc" (UID: "5565d4e3-9657-4dcb-854c-cdfe497459bc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:48.933517 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.933485    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-oauth-config\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:23:48.933517 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.933508    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-serving-cert\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:23:48.933517 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.933518    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgx2x\" (UniqueName: \"kubernetes.io/projected/5565d4e3-9657-4dcb-854c-cdfe497459bc-kube-api-access-xgx2x\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:23:48.933768 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.933527    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-console-config\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:23:48.933768 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.933536    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-oauth-serving-cert\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:23:48.933768 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:48.933544    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5565d4e3-9657-4dcb-854c-cdfe497459bc-service-ca\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:23:49.614998 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:49.614963    2586 generic.go:356] "Generic (PLEG): container finished" podID="5565d4e3-9657-4dcb-854c-cdfe497459bc" containerID="3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb" exitCode=0
Jul 01 19:23:49.615402 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:49.615013    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5c49974-9j957"
Jul 01 19:23:49.615402 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:49.615028    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5c49974-9j957" event={"ID":"5565d4e3-9657-4dcb-854c-cdfe497459bc","Type":"ContainerDied","Data":"3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb"}
Jul 01 19:23:49.615402 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:49.615079    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5c49974-9j957" event={"ID":"5565d4e3-9657-4dcb-854c-cdfe497459bc","Type":"ContainerDied","Data":"22f87442148074e6497431764392b26f3057fdcbbe95623c4a227b90542ed82d"}
Jul 01 19:23:49.615402 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:49.615106    2586 scope.go:117] "RemoveContainer" containerID="3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb"
Jul 01 19:23:49.623115 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:49.623098    2586 scope.go:117] "RemoveContainer" containerID="3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb"
Jul 01 19:23:49.623383 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:23:49.623360    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb\": container with ID starting with 3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb not found: ID does not exist" containerID="3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb"
Jul 01 19:23:49.623465 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:49.623390    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb"} err="failed to get container status \"3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb\": rpc error: code = NotFound desc = could not find container \"3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb\": container with ID starting with 3e6ab09ae91bd8727682003d1ddb4c62cb53b3afbbcb48a36c34fe36a9707bbb not found: ID does not exist"
Jul 01 19:23:49.647208 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:49.647178    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5c49974-9j957"]
Jul 01 19:23:49.648470 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:49.648453    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b5c49974-9j957"]
Jul 01 19:23:49.882402 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:49.882307    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5565d4e3-9657-4dcb-854c-cdfe497459bc" path="/var/lib/kubelet/pods/5565d4e3-9657-4dcb-854c-cdfe497459bc/volumes"
Jul 01 19:23:54.092909 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:54.092866    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:54.158157 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:23:54.158120    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:24:08.311967 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:24:08.311929    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:24:08.633443 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:24:08.633362    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:24:17.880004 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:24:17.879914    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:24:17.980509 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:24:17.980480    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:24:28.968159 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:24:28.968120    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:24:29.069020 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:24:29.068988    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:24:29.135440 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:24:29.135401    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:24:49.195628 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:24:49.195570    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:24:49.255733 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:24:49.255703    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:23.690858 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:23.690821    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:23.743030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:23.742990    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:25.208813 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:25.208778    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:35.669550 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:35.669514    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:35.734646 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:35.734603    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:41.221222 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.221189    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86c5d8b9df-m72x2"]
Jul 01 19:25:41.221709 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.221517    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5565d4e3-9657-4dcb-854c-cdfe497459bc" containerName="console"
Jul 01 19:25:41.221709 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.221536    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5565d4e3-9657-4dcb-854c-cdfe497459bc" containerName="console"
Jul 01 19:25:41.221709 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.221597    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5565d4e3-9657-4dcb-854c-cdfe497459bc" containerName="console"
Jul 01 19:25:41.224485 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.224465    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.227026 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.226997    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 19:25:41.227026 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.226997    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jnpqh\""
Jul 01 19:25:41.227208 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.227028    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 01 19:25:41.227208 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.226999    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 01 19:25:41.227360 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.227342    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 19:25:41.227433 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.227401    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 01 19:25:41.227673 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.227659    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 01 19:25:41.227673 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.227667    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 01 19:25:41.233501 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.233485    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 01 19:25:41.241044 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.241025    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86c5d8b9df-m72x2"]
Jul 01 19:25:41.294598 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.294555    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:41.302642 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.302613    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-oauth-serving-cert\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.302779 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.302646    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-config\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.302779 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.302673    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbltr\" (UniqueName: \"kubernetes.io/projected/0cb2b748-eaf3-4c51-acd8-691594ad984c-kube-api-access-mbltr\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.302779 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.302709    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-serving-cert\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.302779 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.302735    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-trusted-ca-bundle\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.302779 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.302756    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-service-ca\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.302971 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.302845    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-oauth-config\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.395998 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.395963    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:41.404059 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.404026    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-oauth-config\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.404059 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.404061    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-oauth-serving-cert\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.404238 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.404079    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-config\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.404238 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.404103    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbltr\" (UniqueName: \"kubernetes.io/projected/0cb2b748-eaf3-4c51-acd8-691594ad984c-kube-api-access-mbltr\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.404238 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.404130    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-serving-cert\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.404238 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.404183    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-trusted-ca-bundle\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.404238 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.404227    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-service-ca\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.404854 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.404832    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-oauth-serving-cert\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.404964 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.404884    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-config\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.404964 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.404943    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-service-ca\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.405419 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.405396    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-trusted-ca-bundle\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.406535 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.406516    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-oauth-config\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.406658 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.406640    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-serving-cert\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.413990 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.413972    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbltr\" (UniqueName: \"kubernetes.io/projected/0cb2b748-eaf3-4c51-acd8-691594ad984c-kube-api-access-mbltr\") pod \"console-86c5d8b9df-m72x2\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") " pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.500137 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.496473    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:41.534092 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.534050    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:41.677508 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.677484    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86c5d8b9df-m72x2"]
Jul 01 19:25:41.679842 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:25:41.679807    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb2b748_eaf3_4c51_acd8_691594ad984c.slice/crio-fcf0e1cd5c1eba42916f0fbebad4e2cbaf658ca86a5a25d60fb01682d1e4bc53 WatchSource:0}: Error finding container fcf0e1cd5c1eba42916f0fbebad4e2cbaf658ca86a5a25d60fb01682d1e4bc53: Status 404 returned error can't find the container with id fcf0e1cd5c1eba42916f0fbebad4e2cbaf658ca86a5a25d60fb01682d1e4bc53
Jul 01 19:25:41.969481 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.969441    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c5d8b9df-m72x2" event={"ID":"0cb2b748-eaf3-4c51-acd8-691594ad984c","Type":"ContainerStarted","Data":"15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650"}
Jul 01 19:25:41.969655 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.969494    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c5d8b9df-m72x2" event={"ID":"0cb2b748-eaf3-4c51-acd8-691594ad984c","Type":"ContainerStarted","Data":"fcf0e1cd5c1eba42916f0fbebad4e2cbaf658ca86a5a25d60fb01682d1e4bc53"}
Jul 01 19:25:41.993275 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:41.993222    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86c5d8b9df-m72x2" podStartSLOduration=0.993205035 podStartE2EDuration="993.205035ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:25:41.992752673 +0000 UTC m=+256.539002010" watchObservedRunningTime="2026-07-01 19:25:41.993205035 +0000 UTC m=+256.539454371"
Jul 01 19:25:42.025145 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:42.025112    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:42.119803 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:42.119771    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:43.629444 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:43.629414    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:43.724515 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:43.724473    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:51.534968 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:51.534868    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:51.534968 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:51.534934    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:51.539446 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:51.539426    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:52.003766 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:52.003738    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:25:52.183215 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:52.183180    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:52.283377 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:52.283299    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:25:52.383945 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:25:52.383915    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:06.086171 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:06.086129    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:22.908906 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:22.908862    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:22.973949 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:22.973911    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:23.203921 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:23.203879    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:23.298401 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:23.298371    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:23.398514 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:23.398476    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:25.804667 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:25.804641    2586 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 19:26:50.926212 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:50.926169    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:52.665871 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:52.665835    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:56.769076 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:56.769042    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:57.174817 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:57.174709    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:57.380654 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:57.380612    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:57.570339 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:57.570304    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:57.676025 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:57.675985    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:57.771648 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:57.771613    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:57.906188 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:57.906094    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:58.046890 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:58.046851    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:58.311828 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:58.311799    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:58.574742 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:58.574649    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:58.744596 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:58.744564    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:58.817078 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:26:58.817047    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:27:53.980885 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:27:53.980791    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:27:57.605044 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:27:57.604984    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:28:05.025395 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.025358    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:28:05.104652 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.104621    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-bd65f9df8-vdq4n"]
Jul 01 19:28:05.107524 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.107506    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:05.110036 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.110016    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 19:28:05.110151 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.110082    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-9fhc7\""
Jul 01 19:28:05.110959 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.110944    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 19:28:05.111035 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.111022    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 01 19:28:05.133101 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.133077    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-bd65f9df8-vdq4n"]
Jul 01 19:28:05.134068 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.134049    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-6zphh"]
Jul 01 19:28:05.136969 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.136953    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh"
Jul 01 19:28:05.139403 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.139386    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-z2h89\""
Jul 01 19:28:05.139582 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.139570    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 01 19:28:05.146819 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.146792    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:28:05.164625 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.164599    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-6zphh"]
Jul 01 19:28:05.207532 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.207500    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk2f5\" (UniqueName: \"kubernetes.io/projected/79ab8128-aa22-4f98-a8d8-245137660699-kube-api-access-dk2f5\") pod \"kserve-controller-manager-bd65f9df8-vdq4n\" (UID: \"79ab8128-aa22-4f98-a8d8-245137660699\") " pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:05.207671 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.207556    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hkpm\" (UniqueName: \"kubernetes.io/projected/43bc5e3e-2fbb-4775-86b4-1afbdbea5a32-kube-api-access-7hkpm\") pod \"llmisvc-controller-manager-67d95fdc48-6zphh\" (UID: \"43bc5e3e-2fbb-4775-86b4-1afbdbea5a32\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh"
Jul 01 19:28:05.207671 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.207585    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43bc5e3e-2fbb-4775-86b4-1afbdbea5a32-cert\") pod \"llmisvc-controller-manager-67d95fdc48-6zphh\" (UID: \"43bc5e3e-2fbb-4775-86b4-1afbdbea5a32\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh"
Jul 01 19:28:05.207671 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.207609    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79ab8128-aa22-4f98-a8d8-245137660699-cert\") pod \"kserve-controller-manager-bd65f9df8-vdq4n\" (UID: \"79ab8128-aa22-4f98-a8d8-245137660699\") " pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:05.235314 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.235277    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:28:05.308826 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.308743    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hkpm\" (UniqueName: \"kubernetes.io/projected/43bc5e3e-2fbb-4775-86b4-1afbdbea5a32-kube-api-access-7hkpm\") pod \"llmisvc-controller-manager-67d95fdc48-6zphh\" (UID: \"43bc5e3e-2fbb-4775-86b4-1afbdbea5a32\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh"
Jul 01 19:28:05.308826 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.308778    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43bc5e3e-2fbb-4775-86b4-1afbdbea5a32-cert\") pod \"llmisvc-controller-manager-67d95fdc48-6zphh\" (UID: \"43bc5e3e-2fbb-4775-86b4-1afbdbea5a32\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh"
Jul 01 19:28:05.308826 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.308804    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79ab8128-aa22-4f98-a8d8-245137660699-cert\") pod \"kserve-controller-manager-bd65f9df8-vdq4n\" (UID: \"79ab8128-aa22-4f98-a8d8-245137660699\") " pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:05.309038 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.308838    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk2f5\" (UniqueName: \"kubernetes.io/projected/79ab8128-aa22-4f98-a8d8-245137660699-kube-api-access-dk2f5\") pod \"kserve-controller-manager-bd65f9df8-vdq4n\" (UID: \"79ab8128-aa22-4f98-a8d8-245137660699\") " pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:05.311187 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.311163    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43bc5e3e-2fbb-4775-86b4-1afbdbea5a32-cert\") pod \"llmisvc-controller-manager-67d95fdc48-6zphh\" (UID: \"43bc5e3e-2fbb-4775-86b4-1afbdbea5a32\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh"
Jul 01 19:28:05.311317 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.311187    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79ab8128-aa22-4f98-a8d8-245137660699-cert\") pod \"kserve-controller-manager-bd65f9df8-vdq4n\" (UID: \"79ab8128-aa22-4f98-a8d8-245137660699\") " pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:05.325956 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.325928    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk2f5\" (UniqueName: \"kubernetes.io/projected/79ab8128-aa22-4f98-a8d8-245137660699-kube-api-access-dk2f5\") pod \"kserve-controller-manager-bd65f9df8-vdq4n\" (UID: \"79ab8128-aa22-4f98-a8d8-245137660699\") " pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:05.326079 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.326057    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hkpm\" (UniqueName: \"kubernetes.io/projected/43bc5e3e-2fbb-4775-86b4-1afbdbea5a32-kube-api-access-7hkpm\") pod \"llmisvc-controller-manager-67d95fdc48-6zphh\" (UID: \"43bc5e3e-2fbb-4775-86b4-1afbdbea5a32\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh"
Jul 01 19:28:05.379162 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.379123    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:28:05.417288 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.417261    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:05.446205 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.446172    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh"
Jul 01 19:28:05.577243 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:28:05.577212    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ab8128_aa22_4f98_a8d8_245137660699.slice/crio-8b591dc2a2040bc6fd38138c9946106526bacddb7cfed07afb575ee1be8ef60c WatchSource:0}: Error finding container 8b591dc2a2040bc6fd38138c9946106526bacddb7cfed07afb575ee1be8ef60c: Status 404 returned error can't find the container with id 8b591dc2a2040bc6fd38138c9946106526bacddb7cfed07afb575ee1be8ef60c
Jul 01 19:28:05.578477 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.578460    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:28:05.578649 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.578632    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-bd65f9df8-vdq4n"]
Jul 01 19:28:05.583834 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.583814    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-6zphh"]
Jul 01 19:28:05.586806 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:28:05.586786    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod43bc5e3e_2fbb_4775_86b4_1afbdbea5a32.slice/crio-115b563b452d89043d75573c7d3fdc32b34324e135a216dc5c2656b1e9dd14e4 WatchSource:0}: Error finding container 115b563b452d89043d75573c7d3fdc32b34324e135a216dc5c2656b1e9dd14e4: Status 404 returned error can't find the container with id 115b563b452d89043d75573c7d3fdc32b34324e135a216dc5c2656b1e9dd14e4
Jul 01 19:28:05.745641 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.745608    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:28:05.858821 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:05.858737    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:28:06.433034 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:06.432996    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh" event={"ID":"43bc5e3e-2fbb-4775-86b4-1afbdbea5a32","Type":"ContainerStarted","Data":"115b563b452d89043d75573c7d3fdc32b34324e135a216dc5c2656b1e9dd14e4"}
Jul 01 19:28:06.434806 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:06.434773    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n" event={"ID":"79ab8128-aa22-4f98-a8d8-245137660699","Type":"ContainerStarted","Data":"8b591dc2a2040bc6fd38138c9946106526bacddb7cfed07afb575ee1be8ef60c"}
Jul 01 19:28:09.170153 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:09.170052    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:28:09.277782 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:09.277740    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:28:09.448474 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:09.448441    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh" event={"ID":"43bc5e3e-2fbb-4775-86b4-1afbdbea5a32","Type":"ContainerStarted","Data":"2c8a58aee6198627a4d4526ec7e5d7e824020075de64a9b30ad86ec2de7083a0"}
Jul 01 19:28:09.448657 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:09.448551    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh"
Jul 01 19:28:09.449905 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:09.449878    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n" event={"ID":"79ab8128-aa22-4f98-a8d8-245137660699","Type":"ContainerStarted","Data":"63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d"}
Jul 01 19:28:09.450022 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:09.450001    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:09.472176 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:09.472127    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh" podStartSLOduration=1.238602485 podStartE2EDuration="4.472113968s" totalImagesPullingTime="3.233511483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:28:09.470172377 +0000 UTC m=+404.016421713" watchObservedRunningTime="2026-07-01 19:28:09.472113968 +0000 UTC m=+404.018363304"
Jul 01 19:28:40.454917 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:40.454888    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-6zphh"
Jul 01 19:28:40.457855 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:40.457838    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:40.472293 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:40.472245    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n" podStartSLOduration=32.182486797 podStartE2EDuration="35.472229413s" totalImagesPullingTime="3.289742616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:28:09.495863352 +0000 UTC m=+404.042112701" watchObservedRunningTime="2026-07-01 19:28:40.472229413 +0000 UTC m=+435.018478749"
Jul 01 19:28:41.789990 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:41.789956    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-bd65f9df8-vdq4n"]
Jul 01 19:28:41.790423 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:41.790183    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n" podUID="79ab8128-aa22-4f98-a8d8-245137660699" containerName="manager" containerID="cri-o://63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d" gracePeriod=10
Jul 01 19:28:41.814932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:41.814903    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-bd65f9df8-w6r79"]
Jul 01 19:28:41.819158 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:41.819139    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-bd65f9df8-w6r79"
Jul 01 19:28:41.829005 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:41.828975    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-bd65f9df8-w6r79"]
Jul 01 19:28:41.995046 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:41.995008    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j797f\" (UniqueName: \"kubernetes.io/projected/390ef545-bdd8-40c2-b951-0ec703c3c24e-kube-api-access-j797f\") pod \"kserve-controller-manager-bd65f9df8-w6r79\" (UID: \"390ef545-bdd8-40c2-b951-0ec703c3c24e\") " pod="kserve/kserve-controller-manager-bd65f9df8-w6r79"
Jul 01 19:28:41.995192 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:41.995062    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/390ef545-bdd8-40c2-b951-0ec703c3c24e-cert\") pod \"kserve-controller-manager-bd65f9df8-w6r79\" (UID: \"390ef545-bdd8-40c2-b951-0ec703c3c24e\") " pod="kserve/kserve-controller-manager-bd65f9df8-w6r79"
Jul 01 19:28:42.052081 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.052016    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:42.095658 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.095628    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j797f\" (UniqueName: \"kubernetes.io/projected/390ef545-bdd8-40c2-b951-0ec703c3c24e-kube-api-access-j797f\") pod \"kserve-controller-manager-bd65f9df8-w6r79\" (UID: \"390ef545-bdd8-40c2-b951-0ec703c3c24e\") " pod="kserve/kserve-controller-manager-bd65f9df8-w6r79"
Jul 01 19:28:42.095658 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.095662    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/390ef545-bdd8-40c2-b951-0ec703c3c24e-cert\") pod \"kserve-controller-manager-bd65f9df8-w6r79\" (UID: \"390ef545-bdd8-40c2-b951-0ec703c3c24e\") " pod="kserve/kserve-controller-manager-bd65f9df8-w6r79"
Jul 01 19:28:42.097956 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.097935    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/390ef545-bdd8-40c2-b951-0ec703c3c24e-cert\") pod \"kserve-controller-manager-bd65f9df8-w6r79\" (UID: \"390ef545-bdd8-40c2-b951-0ec703c3c24e\") " pod="kserve/kserve-controller-manager-bd65f9df8-w6r79"
Jul 01 19:28:42.105498 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.105468    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j797f\" (UniqueName: \"kubernetes.io/projected/390ef545-bdd8-40c2-b951-0ec703c3c24e-kube-api-access-j797f\") pod \"kserve-controller-manager-bd65f9df8-w6r79\" (UID: \"390ef545-bdd8-40c2-b951-0ec703c3c24e\") " pod="kserve/kserve-controller-manager-bd65f9df8-w6r79"
Jul 01 19:28:42.189625 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.189589    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-bd65f9df8-w6r79"
Jul 01 19:28:42.196871 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.196849    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk2f5\" (UniqueName: \"kubernetes.io/projected/79ab8128-aa22-4f98-a8d8-245137660699-kube-api-access-dk2f5\") pod \"79ab8128-aa22-4f98-a8d8-245137660699\" (UID: \"79ab8128-aa22-4f98-a8d8-245137660699\") "
Jul 01 19:28:42.196932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.196882    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79ab8128-aa22-4f98-a8d8-245137660699-cert\") pod \"79ab8128-aa22-4f98-a8d8-245137660699\" (UID: \"79ab8128-aa22-4f98-a8d8-245137660699\") "
Jul 01 19:28:42.198993 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.198968    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ab8128-aa22-4f98-a8d8-245137660699-kube-api-access-dk2f5" (OuterVolumeSpecName: "kube-api-access-dk2f5") pod "79ab8128-aa22-4f98-a8d8-245137660699" (UID: "79ab8128-aa22-4f98-a8d8-245137660699"). InnerVolumeSpecName "kube-api-access-dk2f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:28:42.199086 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.199021    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ab8128-aa22-4f98-a8d8-245137660699-cert" (OuterVolumeSpecName: "cert") pod "79ab8128-aa22-4f98-a8d8-245137660699" (UID: "79ab8128-aa22-4f98-a8d8-245137660699"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:28:42.298184 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.298154    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk2f5\" (UniqueName: \"kubernetes.io/projected/79ab8128-aa22-4f98-a8d8-245137660699-kube-api-access-dk2f5\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:28:42.298184 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.298182    2586 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79ab8128-aa22-4f98-a8d8-245137660699-cert\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:28:42.341768 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.341745    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-bd65f9df8-w6r79"]
Jul 01 19:28:42.343752 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:28:42.343724    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390ef545_bdd8_40c2_b951_0ec703c3c24e.slice/crio-10b8576621eebcd912e9346021919eff84fd22498c266f3c584c4c3c842f3b2c WatchSource:0}: Error finding container 10b8576621eebcd912e9346021919eff84fd22498c266f3c584c4c3c842f3b2c: Status 404 returned error can't find the container with id 10b8576621eebcd912e9346021919eff84fd22498c266f3c584c4c3c842f3b2c
Jul 01 19:28:42.558436 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.558398    2586 generic.go:356] "Generic (PLEG): container finished" podID="79ab8128-aa22-4f98-a8d8-245137660699" containerID="63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d" exitCode=0
Jul 01 19:28:42.558614 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.558443    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n"
Jul 01 19:28:42.558614 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.558481    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n" event={"ID":"79ab8128-aa22-4f98-a8d8-245137660699","Type":"ContainerDied","Data":"63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d"}
Jul 01 19:28:42.558614 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.558529    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-bd65f9df8-vdq4n" event={"ID":"79ab8128-aa22-4f98-a8d8-245137660699","Type":"ContainerDied","Data":"8b591dc2a2040bc6fd38138c9946106526bacddb7cfed07afb575ee1be8ef60c"}
Jul 01 19:28:42.558614 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.558555    2586 scope.go:117] "RemoveContainer" containerID="63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d"
Jul 01 19:28:42.559819 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.559784    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-bd65f9df8-w6r79" event={"ID":"390ef545-bdd8-40c2-b951-0ec703c3c24e","Type":"ContainerStarted","Data":"10b8576621eebcd912e9346021919eff84fd22498c266f3c584c4c3c842f3b2c"}
Jul 01 19:28:42.566912 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.566895    2586 scope.go:117] "RemoveContainer" containerID="63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d"
Jul 01 19:28:42.567175 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:28:42.567154    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d\": container with ID starting with 63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d not found: ID does not exist" containerID="63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d"
Jul 01 19:28:42.567259 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.567180    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d"} err="failed to get container status \"63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d\": rpc error: code = NotFound desc = could not find container \"63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d\": container with ID starting with 63103667f7e721601fbd46965820f1ffc49dbf77b2082378955f224b9511c48d not found: ID does not exist"
Jul 01 19:28:42.579776 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.579753    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-bd65f9df8-vdq4n"]
Jul 01 19:28:42.586173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:42.586152    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-bd65f9df8-vdq4n"]
Jul 01 19:28:43.568319 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:43.568231    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-bd65f9df8-w6r79" event={"ID":"390ef545-bdd8-40c2-b951-0ec703c3c24e","Type":"ContainerStarted","Data":"a452c707bf568ea6fe245a3b2c558c4084943ddc74cca6333c8dd65eaa2994b1"}
Jul 01 19:28:43.568673 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:43.568341    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-bd65f9df8-w6r79"
Jul 01 19:28:43.586932 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:43.586885    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-bd65f9df8-w6r79" podStartSLOduration=1.7098151270000002 podStartE2EDuration="2.586870896s" totalImagesPullingTime="877.055769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:28:43.585418394 +0000 UTC m=+438.131667730" watchObservedRunningTime="2026-07-01 19:28:43.586870896 +0000 UTC m=+438.133120231"
Jul 01 19:28:43.882446 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:28:43.882371    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ab8128-aa22-4f98-a8d8-245137660699" path="/var/lib/kubelet/pods/79ab8128-aa22-4f98-a8d8-245137660699/volumes"
Jul 01 19:29:14.576843 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:29:14.576811    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-bd65f9df8-w6r79"
Jul 01 19:29:15.674574 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:29:15.674541    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:29:15.779019 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:29:15.778981    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:29:15.851561 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:29:15.851527    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:29:16.660892 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:29:16.660853    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:29:16.730421 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:29:16.730383    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:30:52.235216 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:30:52.235131    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:30:52.317670 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:30:52.317630    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:30:58.123854 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:30:58.123818    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:31:03.637905 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.637878    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:31:03.793078 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.793040    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:31:03.884368 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.884333    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:31:03.960258 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.960223    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:31:03.974176 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.974147    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"]
Jul 01 19:31:03.974397 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.974385    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ab8128-aa22-4f98-a8d8-245137660699" containerName="manager"
Jul 01 19:31:03.974438 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.974399    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ab8128-aa22-4f98-a8d8-245137660699" containerName="manager"
Jul 01 19:31:03.977000 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.975018    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ab8128-aa22-4f98-a8d8-245137660699" containerName="manager"
Jul 01 19:31:03.979435 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.979414    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:03.981894 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.981873    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 01 19:31:03.982034 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.982012    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 01 19:31:03.982108 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.982093    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xs8vf\""
Jul 01 19:31:03.982159 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.982093    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:31:03.982201 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.982092    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:31:03.982859 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.982844    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:31:03.983064 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.983050    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:31:03.988246 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:03.988226    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"]
Jul 01 19:31:04.034534 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.034494    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.034760 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.034557    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.034760 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.034598    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tlhv\" (UniqueName: \"kubernetes.io/projected/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-api-access-5tlhv\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.034760 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.034640    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.034760 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.034699    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.061921 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.061883    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:31:04.135837 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.135797    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.135837 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.135844    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.136091 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.135966    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.136091 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.136005    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlhv\" (UniqueName: \"kubernetes.io/projected/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-api-access-5tlhv\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.136091 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.136043    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.136245 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.136166    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.136699 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.136669    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.136888 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.136868    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.138488 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.138462    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.138870 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.138848    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:31:04.145638 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.145615    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlhv\" (UniqueName: \"kubernetes.io/projected/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-api-access-5tlhv\") pod \"isvc-xgboost-graph-predictor-555fc69d56-zkcnx\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.290895 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.290814    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:04.425381 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.425340    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"]
Jul 01 19:31:04.428120 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:31:04.428091    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e060e8e_2be5_4096_b4f0_7535a1e83d3e.slice/crio-67eba5d0f315fc29f95079a68a001ec789c5964e06abb4f19398a59054cd4cc1 WatchSource:0}: Error finding container 67eba5d0f315fc29f95079a68a001ec789c5964e06abb4f19398a59054cd4cc1: Status 404 returned error can't find the container with id 67eba5d0f315fc29f95079a68a001ec789c5964e06abb4f19398a59054cd4cc1
Jul 01 19:31:04.792661 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.792629    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:31:04.884730 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:04.884667    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:31:05.033862 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:05.033827    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" event={"ID":"3e060e8e-2be5-4096-b4f0-7535a1e83d3e","Type":"ContainerStarted","Data":"67eba5d0f315fc29f95079a68a001ec789c5964e06abb4f19398a59054cd4cc1"}
Jul 01 19:31:08.047503 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:08.047468    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" event={"ID":"3e060e8e-2be5-4096-b4f0-7535a1e83d3e","Type":"ContainerStarted","Data":"068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667"}
Jul 01 19:31:12.061493 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:12.061458    2586 generic.go:356] "Generic (PLEG): container finished" podID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerID="068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667" exitCode=0
Jul 01 19:31:12.061878 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:12.061508    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" event={"ID":"3e060e8e-2be5-4096-b4f0-7535a1e83d3e","Type":"ContainerDied","Data":"068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667"}
Jul 01 19:31:33.146358 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:33.146318    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" event={"ID":"3e060e8e-2be5-4096-b4f0-7535a1e83d3e","Type":"ContainerStarted","Data":"8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82"}
Jul 01 19:31:35.154776 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:35.154738    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" event={"ID":"3e060e8e-2be5-4096-b4f0-7535a1e83d3e","Type":"ContainerStarted","Data":"0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922"}
Jul 01 19:31:35.155161 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:35.154933    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:35.155161 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:35.154975    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:35.156173 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:35.156138    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 19:31:35.177857 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:35.177812    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podStartSLOduration=2.487532431 podStartE2EDuration="32.17779669s" totalImagesPullingTime="26.209700371s" totalInitContainerRuntime="3.480563888s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:31:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:31:35.176387427 +0000 UTC m=+609.722636762" watchObservedRunningTime="2026-07-01 19:31:35.17779669 +0000 UTC m=+609.724046026"
Jul 01 19:31:36.158762 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:36.158724    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 19:31:38.895269 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:38.895232    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:31:41.163489 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:41.163460    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:31:41.164185 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:41.164153    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 19:31:51.165033 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:31:51.164998    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 19:32:01.164238 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:01.164199    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 19:32:11.164433 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:11.164392    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 19:32:21.164184 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:21.164143    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 19:32:23.600540 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:23.600502    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"]
Jul 01 19:32:23.602619 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:23.602600    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:23.604917 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:23.604870    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-79b53-serving-cert\""
Jul 01 19:32:23.604917 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:23.604867    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-79b53-kube-rbac-proxy-sar-config\""
Jul 01 19:32:23.605098 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:23.604942    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:32:23.611233 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:23.611211    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"]
Jul 01 19:32:23.657870 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:23.657836    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44e6f124-7184-4c32-a935-23b84f003d67-openshift-service-ca-bundle\") pod \"switch-graph-79b53-c5dc9c995-jnsgl\" (UID: \"44e6f124-7184-4c32-a935-23b84f003d67\") " pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:23.657870 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:23.657884    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e6f124-7184-4c32-a935-23b84f003d67-proxy-tls\") pod \"switch-graph-79b53-c5dc9c995-jnsgl\" (UID: \"44e6f124-7184-4c32-a935-23b84f003d67\") " pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:23.758276 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:23.758239    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44e6f124-7184-4c32-a935-23b84f003d67-openshift-service-ca-bundle\") pod \"switch-graph-79b53-c5dc9c995-jnsgl\" (UID: \"44e6f124-7184-4c32-a935-23b84f003d67\") " pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:23.758434 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:23.758286    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e6f124-7184-4c32-a935-23b84f003d67-proxy-tls\") pod \"switch-graph-79b53-c5dc9c995-jnsgl\" (UID: \"44e6f124-7184-4c32-a935-23b84f003d67\") " pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:23.758434 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:32:23.758397    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-79b53-serving-cert: secret "switch-graph-79b53-serving-cert" not found
Jul 01 19:32:23.758521 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:32:23.758452    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44e6f124-7184-4c32-a935-23b84f003d67-proxy-tls podName:44e6f124-7184-4c32-a935-23b84f003d67 nodeName:}" failed. No retries permitted until 2026-07-01 19:32:24.258433705 +0000 UTC m=+658.804683023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/44e6f124-7184-4c32-a935-23b84f003d67-proxy-tls") pod "switch-graph-79b53-c5dc9c995-jnsgl" (UID: "44e6f124-7184-4c32-a935-23b84f003d67") : secret "switch-graph-79b53-serving-cert" not found
Jul 01 19:32:23.758892 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:23.758871    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44e6f124-7184-4c32-a935-23b84f003d67-openshift-service-ca-bundle\") pod \"switch-graph-79b53-c5dc9c995-jnsgl\" (UID: \"44e6f124-7184-4c32-a935-23b84f003d67\") " pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:24.262869 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:24.262820    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e6f124-7184-4c32-a935-23b84f003d67-proxy-tls\") pod \"switch-graph-79b53-c5dc9c995-jnsgl\" (UID: \"44e6f124-7184-4c32-a935-23b84f003d67\") " pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:24.265166 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:24.265142    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e6f124-7184-4c32-a935-23b84f003d67-proxy-tls\") pod \"switch-graph-79b53-c5dc9c995-jnsgl\" (UID: \"44e6f124-7184-4c32-a935-23b84f003d67\") " pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:24.513329 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:24.513244    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:24.634010 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:24.633978    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"]
Jul 01 19:32:24.636923 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:32:24.636896    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e6f124_7184_4c32_a935_23b84f003d67.slice/crio-758cf093b503ae7c4693ee58e10b18d9b18d63442a28ed4fba71682f33735f46 WatchSource:0}: Error finding container 758cf093b503ae7c4693ee58e10b18d9b18d63442a28ed4fba71682f33735f46: Status 404 returned error can't find the container with id 758cf093b503ae7c4693ee58e10b18d9b18d63442a28ed4fba71682f33735f46
Jul 01 19:32:25.329160 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:25.329118    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" event={"ID":"44e6f124-7184-4c32-a935-23b84f003d67","Type":"ContainerStarted","Data":"758cf093b503ae7c4693ee58e10b18d9b18d63442a28ed4fba71682f33735f46"}
Jul 01 19:32:25.909891 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:25.909852    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:32:27.338382 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:27.338293    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" event={"ID":"44e6f124-7184-4c32-a935-23b84f003d67","Type":"ContainerStarted","Data":"0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae"}
Jul 01 19:32:27.338745 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:27.338449    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:27.354829 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:27.354775    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" podStartSLOduration=1.953542324 podStartE2EDuration="4.354758702s" totalImagesPullingTime="2.401216378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:32:27.35441588 +0000 UTC m=+661.900665217" watchObservedRunningTime="2026-07-01 19:32:27.354758702 +0000 UTC m=+661.901008038"
Jul 01 19:32:31.164115 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:31.164082    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 01 19:32:33.346899 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:33.346866    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:33.710111 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:33.710082    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"]
Jul 01 19:32:33.710276 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:33.710255    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" podUID="44e6f124-7184-4c32-a935-23b84f003d67" containerName="switch-graph-79b53" containerID="cri-o://0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae" gracePeriod=30
Jul 01 19:32:33.736118 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:33.736076    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:32:33.837279 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:33.837248    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:32:33.919360 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:33.919328    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:32:34.004350 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:34.004265    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:32:34.098953 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:34.098917    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:32:34.248909 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:34.248870    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:32:38.346511 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:38.346466    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" podUID="44e6f124-7184-4c32-a935-23b84f003d67" containerName="switch-graph-79b53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:32:41.165373 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:41.165336    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:32:43.346040 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:43.346002    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" podUID="44e6f124-7184-4c32-a935-23b84f003d67" containerName="switch-graph-79b53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:32:48.346026 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:48.345979    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" podUID="44e6f124-7184-4c32-a935-23b84f003d67" containerName="switch-graph-79b53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:32:48.346387 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:48.346091    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:32:53.346063 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:53.346023    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" podUID="44e6f124-7184-4c32-a935-23b84f003d67" containerName="switch-graph-79b53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:32:58.345585 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:32:58.345539    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" podUID="44e6f124-7184-4c32-a935-23b84f003d67" containerName="switch-graph-79b53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:03.346266 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.346228    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" podUID="44e6f124-7184-4c32-a935-23b84f003d67" containerName="switch-graph-79b53" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:03.624031 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.623934    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"]
Jul 01 19:33:03.630011 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.629975    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:33:03.630133 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.630100    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:03.632545 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.632515    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 01 19:33:03.632545 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.632541    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 01 19:33:03.634870 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.634847    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"]
Jul 01 19:33:03.756920 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.756888    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e268b723-faea-4a65-a7ba-857db63861a3-openshift-service-ca-bundle\") pod \"model-chainer-7988dc58f6-ltwnw\" (UID: \"e268b723-faea-4a65-a7ba-857db63861a3\") " pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:03.756920 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.756927    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e268b723-faea-4a65-a7ba-857db63861a3-proxy-tls\") pod \"model-chainer-7988dc58f6-ltwnw\" (UID: \"e268b723-faea-4a65-a7ba-857db63861a3\") " pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:03.856258 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.856233    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:33:03.857700 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.857669    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e268b723-faea-4a65-a7ba-857db63861a3-openshift-service-ca-bundle\") pod \"model-chainer-7988dc58f6-ltwnw\" (UID: \"e268b723-faea-4a65-a7ba-857db63861a3\") " pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:03.857760 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.857726    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e268b723-faea-4a65-a7ba-857db63861a3-proxy-tls\") pod \"model-chainer-7988dc58f6-ltwnw\" (UID: \"e268b723-faea-4a65-a7ba-857db63861a3\") " pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:03.857852 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:33:03.857840    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 01 19:33:03.857901 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:33:03.857892    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e268b723-faea-4a65-a7ba-857db63861a3-proxy-tls podName:e268b723-faea-4a65-a7ba-857db63861a3 nodeName:}" failed. No retries permitted until 2026-07-01 19:33:04.357876457 +0000 UTC m=+698.904125770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e268b723-faea-4a65-a7ba-857db63861a3-proxy-tls") pod "model-chainer-7988dc58f6-ltwnw" (UID: "e268b723-faea-4a65-a7ba-857db63861a3") : secret "model-chainer-serving-cert" not found
Jul 01 19:33:03.858226 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.858206    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e268b723-faea-4a65-a7ba-857db63861a3-openshift-service-ca-bundle\") pod \"model-chainer-7988dc58f6-ltwnw\" (UID: \"e268b723-faea-4a65-a7ba-857db63861a3\") " pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:03.958055 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.958026    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44e6f124-7184-4c32-a935-23b84f003d67-openshift-service-ca-bundle\") pod \"44e6f124-7184-4c32-a935-23b84f003d67\" (UID: \"44e6f124-7184-4c32-a935-23b84f003d67\") "
Jul 01 19:33:03.958236 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.958060    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e6f124-7184-4c32-a935-23b84f003d67-proxy-tls\") pod \"44e6f124-7184-4c32-a935-23b84f003d67\" (UID: \"44e6f124-7184-4c32-a935-23b84f003d67\") "
Jul 01 19:33:03.958408 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.958386    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e6f124-7184-4c32-a935-23b84f003d67-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "44e6f124-7184-4c32-a935-23b84f003d67" (UID: "44e6f124-7184-4c32-a935-23b84f003d67"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:33:03.960213 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.960190    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e6f124-7184-4c32-a935-23b84f003d67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44e6f124-7184-4c32-a935-23b84f003d67" (UID: "44e6f124-7184-4c32-a935-23b84f003d67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:33:03.998398 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:03.998361    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:33:04.059401 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.059356    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44e6f124-7184-4c32-a935-23b84f003d67-openshift-service-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:33:04.059401 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.059396    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44e6f124-7184-4c32-a935-23b84f003d67-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:33:04.361824 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.361741    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e268b723-faea-4a65-a7ba-857db63861a3-proxy-tls\") pod \"model-chainer-7988dc58f6-ltwnw\" (UID: \"e268b723-faea-4a65-a7ba-857db63861a3\") " pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:04.364039 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.364020    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e268b723-faea-4a65-a7ba-857db63861a3-proxy-tls\") pod \"model-chainer-7988dc58f6-ltwnw\" (UID: \"e268b723-faea-4a65-a7ba-857db63861a3\") " pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:04.465375 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.465338    2586 generic.go:356] "Generic (PLEG): container finished" podID="44e6f124-7184-4c32-a935-23b84f003d67" containerID="0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae" exitCode=0
Jul 01 19:33:04.465530 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.465392    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"
Jul 01 19:33:04.465530 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.465423    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" event={"ID":"44e6f124-7184-4c32-a935-23b84f003d67","Type":"ContainerDied","Data":"0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae"}
Jul 01 19:33:04.465530 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.465477    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl" event={"ID":"44e6f124-7184-4c32-a935-23b84f003d67","Type":"ContainerDied","Data":"758cf093b503ae7c4693ee58e10b18d9b18d63442a28ed4fba71682f33735f46"}
Jul 01 19:33:04.465530 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.465501    2586 scope.go:117] "RemoveContainer" containerID="0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae"
Jul 01 19:33:04.473380 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.473364    2586 scope.go:117] "RemoveContainer" containerID="0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae"
Jul 01 19:33:04.473623 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:33:04.473606    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae\": container with ID starting with 0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae not found: ID does not exist" containerID="0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae"
Jul 01 19:33:04.473708 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.473630    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae"} err="failed to get container status \"0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae\": rpc error: code = NotFound desc = could not find container \"0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae\": container with ID starting with 0d9ce8c025b4b94a00c76790b65fd1614588a6615b5f4486c9b7986ead7cb6ae not found: ID does not exist"
Jul 01 19:33:04.485763 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.485739    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"]
Jul 01 19:33:04.487431 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.487406    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-79b53-c5dc9c995-jnsgl"]
Jul 01 19:33:04.540850 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.540824    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:04.664749 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:04.664593    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"]
Jul 01 19:33:04.667090 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:33:04.667052    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode268b723_faea_4a65_a7ba_857db63861a3.slice/crio-994991dbaf230367a5b610f3fa5c4833cef7ca446582370d8fe0b30c0bd297c1 WatchSource:0}: Error finding container 994991dbaf230367a5b610f3fa5c4833cef7ca446582370d8fe0b30c0bd297c1: Status 404 returned error can't find the container with id 994991dbaf230367a5b610f3fa5c4833cef7ca446582370d8fe0b30c0bd297c1
Jul 01 19:33:05.471281 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:05.471246    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" event={"ID":"e268b723-faea-4a65-a7ba-857db63861a3","Type":"ContainerStarted","Data":"0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128"}
Jul 01 19:33:05.471677 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:05.471299    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" event={"ID":"e268b723-faea-4a65-a7ba-857db63861a3","Type":"ContainerStarted","Data":"994991dbaf230367a5b610f3fa5c4833cef7ca446582370d8fe0b30c0bd297c1"}
Jul 01 19:33:05.471677 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:05.471339    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:05.490245 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:05.490197    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" podStartSLOduration=2.490182268 podStartE2EDuration="2.490182268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:33:05.488805577 +0000 UTC m=+700.035054925" watchObservedRunningTime="2026-07-01 19:33:05.490182268 +0000 UTC m=+700.036431603"
Jul 01 19:33:05.882702 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:05.882610    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44e6f124-7184-4c32-a935-23b84f003d67" path="/var/lib/kubelet/pods/44e6f124-7184-4c32-a935-23b84f003d67/volumes"
Jul 01 19:33:11.480452 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:11.480425    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:13.738402 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:13.738372    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"]
Jul 01 19:33:13.738805 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:13.738540    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" podUID="e268b723-faea-4a65-a7ba-857db63861a3" containerName="model-chainer" containerID="cri-o://0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128" gracePeriod=30
Jul 01 19:33:13.778990 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:13.778962    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:33:13.831250 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:13.831219    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"]
Jul 01 19:33:13.831532 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:13.831504    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container" containerID="cri-o://8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82" gracePeriod=30
Jul 01 19:33:13.831613 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:13.831532    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kube-rbac-proxy" containerID="cri-o://0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922" gracePeriod=30
Jul 01 19:33:13.868832 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:13.868801    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:33:13.959954 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:13.959923    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:33:14.108936 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:14.108843    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:33:14.210135 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:14.210096    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:33:14.506151 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:14.506119    2586 generic.go:356] "Generic (PLEG): container finished" podID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerID="0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922" exitCode=2
Jul 01 19:33:14.506381 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:14.506168    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" event={"ID":"3e060e8e-2be5-4096-b4f0-7535a1e83d3e","Type":"ContainerDied","Data":"0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922"}
Jul 01 19:33:16.159127 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:16.159085    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jul 01 19:33:16.479630 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:16.479596    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" podUID="e268b723-faea-4a65-a7ba-857db63861a3" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:17.484875 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.484850    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:33:17.519340 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.519247    2586 generic.go:356] "Generic (PLEG): container finished" podID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerID="8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82" exitCode=0
Jul 01 19:33:17.519340 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.519309    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"
Jul 01 19:33:17.519546 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.519328    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" event={"ID":"3e060e8e-2be5-4096-b4f0-7535a1e83d3e","Type":"ContainerDied","Data":"8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82"}
Jul 01 19:33:17.519546 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.519406    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx" event={"ID":"3e060e8e-2be5-4096-b4f0-7535a1e83d3e","Type":"ContainerDied","Data":"67eba5d0f315fc29f95079a68a001ec789c5964e06abb4f19398a59054cd4cc1"}
Jul 01 19:33:17.519546 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.519444    2586 scope.go:117] "RemoveContainer" containerID="0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922"
Jul 01 19:33:17.527054 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.527033    2586 scope.go:117] "RemoveContainer" containerID="8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82"
Jul 01 19:33:17.533820 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.533802    2586 scope.go:117] "RemoveContainer" containerID="068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667"
Jul 01 19:33:17.540808 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.540789    2586 scope.go:117] "RemoveContainer" containerID="0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922"
Jul 01 19:33:17.541076 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:33:17.541056    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922\": container with ID starting with 0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922 not found: ID does not exist" containerID="0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922"
Jul 01 19:33:17.541139 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.541085    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922"} err="failed to get container status \"0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922\": rpc error: code = NotFound desc = could not find container \"0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922\": container with ID starting with 0e9aae6fddb535ef988b32e3d0d0ab093202ca4d1c5847170f878e77d5c48922 not found: ID does not exist"
Jul 01 19:33:17.541139 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.541105    2586 scope.go:117] "RemoveContainer" containerID="8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82"
Jul 01 19:33:17.541340 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:33:17.541323    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82\": container with ID starting with 8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82 not found: ID does not exist" containerID="8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82"
Jul 01 19:33:17.541382 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.541346    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82"} err="failed to get container status \"8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82\": rpc error: code = NotFound desc = could not find container \"8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82\": container with ID starting with 8f28f0c670d9ccf5e5b71c545fa64655b48435a6a8ab56dae2a9889e590aff82 not found: ID does not exist"
Jul 01 19:33:17.541382 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.541359    2586 scope.go:117] "RemoveContainer" containerID="068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667"
Jul 01 19:33:17.541562 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:33:17.541544    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667\": container with ID starting with 068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667 not found: ID does not exist" containerID="068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667"
Jul 01 19:33:17.541601 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.541570    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667"} err="failed to get container status \"068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667\": rpc error: code = NotFound desc = could not find container \"068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667\": container with ID starting with 068bcdeb0358a4eedba770751000c348fa2fccc037366294f2c6ba7f0d6ad667 not found: ID does not exist"
Jul 01 19:33:17.569878 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.569848    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-rbac-proxy-sar-config\") pod \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") "
Jul 01 19:33:17.570004 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.569885    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kserve-provision-location\") pod \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") "
Jul 01 19:33:17.570004 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.569930    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-cabundle-cert\") pod \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") "
Jul 01 19:33:17.570004 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.569974    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-proxy-tls\") pod \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") "
Jul 01 19:33:17.570163 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.570005    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tlhv\" (UniqueName: \"kubernetes.io/projected/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-api-access-5tlhv\") pod \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\" (UID: \"3e060e8e-2be5-4096-b4f0-7535a1e83d3e\") "
Jul 01 19:33:17.570258 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.570236    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e060e8e-2be5-4096-b4f0-7535a1e83d3e" (UID: "3e060e8e-2be5-4096-b4f0-7535a1e83d3e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:33:17.570320 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.570259    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e060e8e-2be5-4096-b4f0-7535a1e83d3e" (UID: "3e060e8e-2be5-4096-b4f0-7535a1e83d3e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:33:17.570370 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.570317    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e060e8e-2be5-4096-b4f0-7535a1e83d3e" (UID: "3e060e8e-2be5-4096-b4f0-7535a1e83d3e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:33:17.572045 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.572023    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-api-access-5tlhv" (OuterVolumeSpecName: "kube-api-access-5tlhv") pod "3e060e8e-2be5-4096-b4f0-7535a1e83d3e" (UID: "3e060e8e-2be5-4096-b4f0-7535a1e83d3e"). InnerVolumeSpecName "kube-api-access-5tlhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:33:17.572134 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.572115    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e060e8e-2be5-4096-b4f0-7535a1e83d3e" (UID: "3e060e8e-2be5-4096-b4f0-7535a1e83d3e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:33:17.671295 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.671254    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tlhv\" (UniqueName: \"kubernetes.io/projected/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-api-access-5tlhv\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:33:17.671295 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.671287    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:33:17.671295 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.671299    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-kserve-provision-location\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:33:17.671521 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.671311    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-cabundle-cert\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:33:17.671521 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.671325    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e060e8e-2be5-4096-b4f0-7535a1e83d3e-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:33:17.842326 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.842295    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"]
Jul 01 19:33:17.844548 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.844518    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-zkcnx"]
Jul 01 19:33:17.883436 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:17.883406    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" path="/var/lib/kubelet/pods/3e060e8e-2be5-4096-b4f0-7535a1e83d3e/volumes"
Jul 01 19:33:21.478863 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:21.478817    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" podUID="e268b723-faea-4a65-a7ba-857db63861a3" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:26.479458 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:26.479423    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" podUID="e268b723-faea-4a65-a7ba-857db63861a3" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:26.479901 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:26.479524    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:31.478841 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:31.478794    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" podUID="e268b723-faea-4a65-a7ba-857db63861a3" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:33.939769 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.939729    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"]
Jul 01 19:33:33.940210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.940086    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e6f124-7184-4c32-a935-23b84f003d67" containerName="switch-graph-79b53"
Jul 01 19:33:33.940210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.940103    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e6f124-7184-4c32-a935-23b84f003d67" containerName="switch-graph-79b53"
Jul 01 19:33:33.940210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.940120    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container"
Jul 01 19:33:33.940210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.940128    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container"
Jul 01 19:33:33.940210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.940138    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="storage-initializer"
Jul 01 19:33:33.940210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.940146    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="storage-initializer"
Jul 01 19:33:33.940210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.940165    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kube-rbac-proxy"
Jul 01 19:33:33.940210 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.940172    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kube-rbac-proxy"
Jul 01 19:33:33.940588 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.940233    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kserve-container"
Jul 01 19:33:33.940588 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.940246    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e6f124-7184-4c32-a935-23b84f003d67" containerName="switch-graph-79b53"
Jul 01 19:33:33.940588 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.940256    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e060e8e-2be5-4096-b4f0-7535a1e83d3e" containerName="kube-rbac-proxy"
Jul 01 19:33:33.943500 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.943470    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:33:33.944388 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.944365    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:33:33.946577 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.946561    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-6c6c1-serving-cert\""
Jul 01 19:33:33.946671 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.946563    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-6c6c1-kube-rbac-proxy-sar-config\""
Jul 01 19:33:33.950927 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.950821    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"]
Jul 01 19:33:33.992747 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.992713    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795cf749-d561-441f-ade3-9ad1ae0f5c08-proxy-tls\") pod \"switch-graph-6c6c1-578d4456d5-mfhwn\" (UID: \"795cf749-d561-441f-ade3-9ad1ae0f5c08\") " pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:33:33.992747 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:33.992751    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795cf749-d561-441f-ade3-9ad1ae0f5c08-openshift-service-ca-bundle\") pod \"switch-graph-6c6c1-578d4456d5-mfhwn\" (UID: \"795cf749-d561-441f-ade3-9ad1ae0f5c08\") " pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:33:34.093347 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:34.093311    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795cf749-d561-441f-ade3-9ad1ae0f5c08-proxy-tls\") pod \"switch-graph-6c6c1-578d4456d5-mfhwn\" (UID: \"795cf749-d561-441f-ade3-9ad1ae0f5c08\") " pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:33:34.093508 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:34.093357    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795cf749-d561-441f-ade3-9ad1ae0f5c08-openshift-service-ca-bundle\") pod \"switch-graph-6c6c1-578d4456d5-mfhwn\" (UID: \"795cf749-d561-441f-ade3-9ad1ae0f5c08\") " pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:33:34.093508 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:33:34.093460    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-6c6c1-serving-cert: secret "switch-graph-6c6c1-serving-cert" not found
Jul 01 19:33:34.093584 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:33:34.093537    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795cf749-d561-441f-ade3-9ad1ae0f5c08-proxy-tls podName:795cf749-d561-441f-ade3-9ad1ae0f5c08 nodeName:}" failed. No retries permitted until 2026-07-01 19:33:34.593518982 +0000 UTC m=+729.139768300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/795cf749-d561-441f-ade3-9ad1ae0f5c08-proxy-tls") pod "switch-graph-6c6c1-578d4456d5-mfhwn" (UID: "795cf749-d561-441f-ade3-9ad1ae0f5c08") : secret "switch-graph-6c6c1-serving-cert" not found
Jul 01 19:33:34.094028 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:34.094008    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795cf749-d561-441f-ade3-9ad1ae0f5c08-openshift-service-ca-bundle\") pod \"switch-graph-6c6c1-578d4456d5-mfhwn\" (UID: \"795cf749-d561-441f-ade3-9ad1ae0f5c08\") " pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:33:34.311389 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:34.311352    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:33:34.596490 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:34.596410    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795cf749-d561-441f-ade3-9ad1ae0f5c08-proxy-tls\") pod \"switch-graph-6c6c1-578d4456d5-mfhwn\" (UID: \"795cf749-d561-441f-ade3-9ad1ae0f5c08\") " pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:33:34.598882 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:34.598851    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795cf749-d561-441f-ade3-9ad1ae0f5c08-proxy-tls\") pod \"switch-graph-6c6c1-578d4456d5-mfhwn\" (UID: \"795cf749-d561-441f-ade3-9ad1ae0f5c08\") " pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:33:34.854557 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:34.854475    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:33:34.977733 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:34.977703    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"]
Jul 01 19:33:34.980575 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:33:34.980548    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795cf749_d561_441f_ade3_9ad1ae0f5c08.slice/crio-531fdd91a79666105ed2de52cf88fa5617544df1c908fe99bc4431d66cdd00db WatchSource:0}: Error finding container 531fdd91a79666105ed2de52cf88fa5617544df1c908fe99bc4431d66cdd00db: Status 404 returned error can't find the container with id 531fdd91a79666105ed2de52cf88fa5617544df1c908fe99bc4431d66cdd00db
Jul 01 19:33:35.584152 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:35.584115    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" event={"ID":"795cf749-d561-441f-ade3-9ad1ae0f5c08","Type":"ContainerStarted","Data":"89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07"}
Jul 01 19:33:35.584322 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:35.584170    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" event={"ID":"795cf749-d561-441f-ade3-9ad1ae0f5c08","Type":"ContainerStarted","Data":"531fdd91a79666105ed2de52cf88fa5617544df1c908fe99bc4431d66cdd00db"}
Jul 01 19:33:35.584322 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:35.584223    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:33:35.599307 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:35.599265    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" podStartSLOduration=2.599249328 podStartE2EDuration="2.599249328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:33:35.598726416 +0000 UTC m=+730.144975753" watchObservedRunningTime="2026-07-01 19:33:35.599249328 +0000 UTC m=+730.145498703"
Jul 01 19:33:36.479441 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:36.479406    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" podUID="e268b723-faea-4a65-a7ba-857db63861a3" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:41.479012 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:41.478977    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" podUID="e268b723-faea-4a65-a7ba-857db63861a3" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:41.592657 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:41.592629    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:33:43.878479 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:43.878447    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:43.979134 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:43.979101    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e268b723-faea-4a65-a7ba-857db63861a3-openshift-service-ca-bundle\") pod \"e268b723-faea-4a65-a7ba-857db63861a3\" (UID: \"e268b723-faea-4a65-a7ba-857db63861a3\") "
Jul 01 19:33:43.979295 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:43.979182    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e268b723-faea-4a65-a7ba-857db63861a3-proxy-tls\") pod \"e268b723-faea-4a65-a7ba-857db63861a3\" (UID: \"e268b723-faea-4a65-a7ba-857db63861a3\") "
Jul 01 19:33:43.979506 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:43.979486    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e268b723-faea-4a65-a7ba-857db63861a3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e268b723-faea-4a65-a7ba-857db63861a3" (UID: "e268b723-faea-4a65-a7ba-857db63861a3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:33:43.981249 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:43.981228    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e268b723-faea-4a65-a7ba-857db63861a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e268b723-faea-4a65-a7ba-857db63861a3" (UID: "e268b723-faea-4a65-a7ba-857db63861a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:33:44.079744 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:44.079658    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e268b723-faea-4a65-a7ba-857db63861a3-openshift-service-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:33:44.079744 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:44.079714    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e268b723-faea-4a65-a7ba-857db63861a3-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:33:44.615228 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:44.615195    2586 generic.go:356] "Generic (PLEG): container finished" podID="e268b723-faea-4a65-a7ba-857db63861a3" containerID="0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128" exitCode=0
Jul 01 19:33:44.615376 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:44.615240    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"
Jul 01 19:33:44.615376 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:44.615264    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" event={"ID":"e268b723-faea-4a65-a7ba-857db63861a3","Type":"ContainerDied","Data":"0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128"}
Jul 01 19:33:44.615376 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:44.615306    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw" event={"ID":"e268b723-faea-4a65-a7ba-857db63861a3","Type":"ContainerDied","Data":"994991dbaf230367a5b610f3fa5c4833cef7ca446582370d8fe0b30c0bd297c1"}
Jul 01 19:33:44.615376 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:44.615328    2586 scope.go:117] "RemoveContainer" containerID="0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128"
Jul 01 19:33:44.623498 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:44.623481    2586 scope.go:117] "RemoveContainer" containerID="0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128"
Jul 01 19:33:44.623756 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:33:44.623738    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128\": container with ID starting with 0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128 not found: ID does not exist" containerID="0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128"
Jul 01 19:33:44.623807 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:44.623765    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128"} err="failed to get container status \"0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128\": rpc error: code = NotFound desc = could not find container \"0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128\": container with ID starting with 0607f4ecd0865d93fed25df5918044c8f2350ab8a59e2bde68836a6200b74128 not found: ID does not exist"
Jul 01 19:33:44.635887 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:44.635863    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"]
Jul 01 19:33:44.640250 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:44.640227    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7988dc58f6-ltwnw"]
Jul 01 19:33:45.882750 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:33:45.882721    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e268b723-faea-4a65-a7ba-857db63861a3" path="/var/lib/kubelet/pods/e268b723-faea-4a65-a7ba-857db63861a3/volumes"
Jul 01 19:34:13.907328 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:13.907297    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"]
Jul 01 19:34:13.907740 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:13.907554    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e268b723-faea-4a65-a7ba-857db63861a3" containerName="model-chainer"
Jul 01 19:34:13.907740 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:13.907567    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e268b723-faea-4a65-a7ba-857db63861a3" containerName="model-chainer"
Jul 01 19:34:13.907740 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:13.907625    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e268b723-faea-4a65-a7ba-857db63861a3" containerName="model-chainer"
Jul 01 19:34:13.910614 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:13.910596    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:34:13.912950 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:13.912922    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-d109c-serving-cert\""
Jul 01 19:34:13.913044 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:13.912962    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-d109c-kube-rbac-proxy-sar-config\""
Jul 01 19:34:13.921379 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:13.921357    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"]
Jul 01 19:34:13.935491 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:13.935462    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:34:14.003577 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:14.003536    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eed7b11-4b52-4838-a9b3-103c34b46928-openshift-service-ca-bundle\") pod \"sequence-graph-d109c-6ddb5c8b58-56ppr\" (UID: \"0eed7b11-4b52-4838-a9b3-103c34b46928\") " pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:34:14.003762 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:14.003605    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eed7b11-4b52-4838-a9b3-103c34b46928-proxy-tls\") pod \"sequence-graph-d109c-6ddb5c8b58-56ppr\" (UID: \"0eed7b11-4b52-4838-a9b3-103c34b46928\") " pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:34:14.104923 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:14.104883    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eed7b11-4b52-4838-a9b3-103c34b46928-openshift-service-ca-bundle\") pod \"sequence-graph-d109c-6ddb5c8b58-56ppr\" (UID: \"0eed7b11-4b52-4838-a9b3-103c34b46928\") " pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:34:14.105071 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:14.104956    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eed7b11-4b52-4838-a9b3-103c34b46928-proxy-tls\") pod \"sequence-graph-d109c-6ddb5c8b58-56ppr\" (UID: \"0eed7b11-4b52-4838-a9b3-103c34b46928\") " pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:34:14.105071 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:34:14.105054    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-d109c-serving-cert: secret "sequence-graph-d109c-serving-cert" not found
Jul 01 19:34:14.105154 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:34:14.105119    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0eed7b11-4b52-4838-a9b3-103c34b46928-proxy-tls podName:0eed7b11-4b52-4838-a9b3-103c34b46928 nodeName:}" failed. No retries permitted until 2026-07-01 19:34:14.605102284 +0000 UTC m=+769.151351597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0eed7b11-4b52-4838-a9b3-103c34b46928-proxy-tls") pod "sequence-graph-d109c-6ddb5c8b58-56ppr" (UID: "0eed7b11-4b52-4838-a9b3-103c34b46928") : secret "sequence-graph-d109c-serving-cert" not found
Jul 01 19:34:14.105533 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:14.105512    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eed7b11-4b52-4838-a9b3-103c34b46928-openshift-service-ca-bundle\") pod \"sequence-graph-d109c-6ddb5c8b58-56ppr\" (UID: \"0eed7b11-4b52-4838-a9b3-103c34b46928\") " pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:34:14.313488 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:14.313457    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:34:14.610651 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:14.610563    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eed7b11-4b52-4838-a9b3-103c34b46928-proxy-tls\") pod \"sequence-graph-d109c-6ddb5c8b58-56ppr\" (UID: \"0eed7b11-4b52-4838-a9b3-103c34b46928\") " pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:34:14.613043 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:14.613019    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eed7b11-4b52-4838-a9b3-103c34b46928-proxy-tls\") pod \"sequence-graph-d109c-6ddb5c8b58-56ppr\" (UID: \"0eed7b11-4b52-4838-a9b3-103c34b46928\") " pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:34:14.821137 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:14.821099    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:34:14.945287 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:14.945254    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"]
Jul 01 19:34:14.948419 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:34:14.948391    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eed7b11_4b52_4838_a9b3_103c34b46928.slice/crio-8c4642254dedd8a11791bc7c71628c790fecf0da678b2b54bbf0f87b73c3c8a1 WatchSource:0}: Error finding container 8c4642254dedd8a11791bc7c71628c790fecf0da678b2b54bbf0f87b73c3c8a1: Status 404 returned error can't find the container with id 8c4642254dedd8a11791bc7c71628c790fecf0da678b2b54bbf0f87b73c3c8a1
Jul 01 19:34:15.721564 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:15.721529    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" event={"ID":"0eed7b11-4b52-4838-a9b3-103c34b46928","Type":"ContainerStarted","Data":"6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab"}
Jul 01 19:34:15.721751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:15.721578    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" event={"ID":"0eed7b11-4b52-4838-a9b3-103c34b46928","Type":"ContainerStarted","Data":"8c4642254dedd8a11791bc7c71628c790fecf0da678b2b54bbf0f87b73c3c8a1"}
Jul 01 19:34:15.721751 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:15.721625    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:34:15.736917 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:15.736877    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" podStartSLOduration=2.7368641179999997 podStartE2EDuration="2.736864118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:34:15.735385334 +0000 UTC m=+770.281634684" watchObservedRunningTime="2026-07-01 19:34:15.736864118 +0000 UTC m=+770.283113454"
Jul 01 19:34:21.733275 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:21.733242    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:34:31.919531 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:31.919497    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d77d55d66-dsjjk"]
Jul 01 19:34:31.923771 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:31.923749    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:31.933770 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:31.933744    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d77d55d66-dsjjk"]
Jul 01 19:34:31.987487 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:31.987443    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:34:32.035756 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.035713    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-console-oauth-config\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.035756 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.035757    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-console-serving-cert\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.035990 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.035877    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-oauth-serving-cert\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.035990 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.035907    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrrvz\" (UniqueName: \"kubernetes.io/projected/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-kube-api-access-vrrvz\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.035990 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.035958    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-console-config\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.035990 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.035982    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-trusted-ca-bundle\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.036135 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.036012    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-service-ca\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.088084 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.088055    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:34:32.137182 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.137141    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-console-oauth-config\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.137182 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.137187    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-console-serving-cert\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.137403 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.137230    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-oauth-serving-cert\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.137403 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.137247    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrrvz\" (UniqueName: \"kubernetes.io/projected/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-kube-api-access-vrrvz\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.137403 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.137270    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-console-config\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.137403 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.137287    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-trusted-ca-bundle\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.137403 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.137353    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-service-ca\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.138144 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.138117    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-console-config\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.138246 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.138140    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-service-ca\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.138246 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.138127    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-oauth-serving-cert\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.138246 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.138199    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-trusted-ca-bundle\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.139672 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.139658    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-console-oauth-config\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.139794 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.139777    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-console-serving-cert\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.145712 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.145645    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrrvz\" (UniqueName: \"kubernetes.io/projected/3ca503e9-7c14-4560-9ec6-85d11b89e2bc-kube-api-access-vrrvz\") pod \"console-7d77d55d66-dsjjk\" (UID: \"3ca503e9-7c14-4560-9ec6-85d11b89e2bc\") " pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.146561 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.146540    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:34:32.232963 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.232933    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:32.365974 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.365950    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d77d55d66-dsjjk"]
Jul 01 19:34:32.368340 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:34:32.368314    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca503e9_7c14_4560_9ec6_85d11b89e2bc.slice/crio-a4018b6becbd03a5f8641345464b39c459e86876c08a03cbb0837796e244fd49 WatchSource:0}: Error finding container a4018b6becbd03a5f8641345464b39c459e86876c08a03cbb0837796e244fd49: Status 404 returned error can't find the container with id a4018b6becbd03a5f8641345464b39c459e86876c08a03cbb0837796e244fd49
Jul 01 19:34:32.713611 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.713583    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:34:32.784651 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.784619    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d77d55d66-dsjjk" event={"ID":"3ca503e9-7c14-4560-9ec6-85d11b89e2bc","Type":"ContainerStarted","Data":"f21e86e026566351b14788af7caf68d866b5c619479bc09c441325f23d44e4c4"}
Jul 01 19:34:32.784861 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.784674    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d77d55d66-dsjjk" event={"ID":"3ca503e9-7c14-4560-9ec6-85d11b89e2bc","Type":"ContainerStarted","Data":"a4018b6becbd03a5f8641345464b39c459e86876c08a03cbb0837796e244fd49"}
Jul 01 19:34:32.801155 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.801034    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d77d55d66-dsjjk" podStartSLOduration=1.801017888 podStartE2EDuration="1.801017888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:34:32.800388095 +0000 UTC m=+787.346637432" watchObservedRunningTime="2026-07-01 19:34:32.801017888 +0000 UTC m=+787.347267223"
Jul 01 19:34:32.805247 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:32.805221    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:34:42.233395 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:42.233340    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:42.233825 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:42.233417    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:42.238010 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:42.237986    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:42.828326 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:42.828299    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d77d55d66-dsjjk"
Jul 01 19:34:42.876314 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:42.876283    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86c5d8b9df-m72x2"]
Jul 01 19:34:42.996350 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:42.996316    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:34:43.064035 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:43.063997    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:34:43.134091 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:34:43.134011    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:35:07.896001 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:07.895946    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-86c5d8b9df-m72x2" podUID="0cb2b748-eaf3-4c51-acd8-691594ad984c" containerName="console" containerID="cri-o://15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650" gracePeriod=15
Jul 01 19:35:08.145407 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.145383    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:35:08.203987 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.203964    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-config\") pod \"0cb2b748-eaf3-4c51-acd8-691594ad984c\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") "
Jul 01 19:35:08.204172 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.204003    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbltr\" (UniqueName: \"kubernetes.io/projected/0cb2b748-eaf3-4c51-acd8-691594ad984c-kube-api-access-mbltr\") pod \"0cb2b748-eaf3-4c51-acd8-691594ad984c\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") "
Jul 01 19:35:08.204172 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.204031    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-serving-cert\") pod \"0cb2b748-eaf3-4c51-acd8-691594ad984c\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") "
Jul 01 19:35:08.204172 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.204046    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-trusted-ca-bundle\") pod \"0cb2b748-eaf3-4c51-acd8-691594ad984c\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") "
Jul 01 19:35:08.204172 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.204070    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-oauth-serving-cert\") pod \"0cb2b748-eaf3-4c51-acd8-691594ad984c\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") "
Jul 01 19:35:08.204380 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.204241    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-service-ca\") pod \"0cb2b748-eaf3-4c51-acd8-691594ad984c\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") "
Jul 01 19:35:08.204380 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.204287    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-oauth-config\") pod \"0cb2b748-eaf3-4c51-acd8-691594ad984c\" (UID: \"0cb2b748-eaf3-4c51-acd8-691594ad984c\") "
Jul 01 19:35:08.204509 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.204476    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-config" (OuterVolumeSpecName: "console-config") pod "0cb2b748-eaf3-4c51-acd8-691594ad984c" (UID: "0cb2b748-eaf3-4c51-acd8-691594ad984c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:35:08.204509 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.204479    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0cb2b748-eaf3-4c51-acd8-691594ad984c" (UID: "0cb2b748-eaf3-4c51-acd8-691594ad984c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:35:08.204617 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.204513    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0cb2b748-eaf3-4c51-acd8-691594ad984c" (UID: "0cb2b748-eaf3-4c51-acd8-691594ad984c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:35:08.204617 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.204593    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-service-ca" (OuterVolumeSpecName: "service-ca") pod "0cb2b748-eaf3-4c51-acd8-691594ad984c" (UID: "0cb2b748-eaf3-4c51-acd8-691594ad984c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:35:08.206309 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.206284    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0cb2b748-eaf3-4c51-acd8-691594ad984c" (UID: "0cb2b748-eaf3-4c51-acd8-691594ad984c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:35:08.206408 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.206323    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0cb2b748-eaf3-4c51-acd8-691594ad984c" (UID: "0cb2b748-eaf3-4c51-acd8-691594ad984c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:35:08.206408 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.206329    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb2b748-eaf3-4c51-acd8-691594ad984c-kube-api-access-mbltr" (OuterVolumeSpecName: "kube-api-access-mbltr") pod "0cb2b748-eaf3-4c51-acd8-691594ad984c" (UID: "0cb2b748-eaf3-4c51-acd8-691594ad984c"). InnerVolumeSpecName "kube-api-access-mbltr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:35:08.305074 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.305027    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-config\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:35:08.305074 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.305066    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbltr\" (UniqueName: \"kubernetes.io/projected/0cb2b748-eaf3-4c51-acd8-691594ad984c-kube-api-access-mbltr\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:35:08.305074 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.305076    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-serving-cert\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:35:08.305074 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.305087    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-trusted-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:35:08.305336 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.305096    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-oauth-serving-cert\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:35:08.305336 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.305104    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2b748-eaf3-4c51-acd8-691594ad984c-service-ca\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:35:08.305336 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.305112    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb2b748-eaf3-4c51-acd8-691594ad984c-console-oauth-config\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:35:08.914800 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.914766    2586 generic.go:356] "Generic (PLEG): container finished" podID="0cb2b748-eaf3-4c51-acd8-691594ad984c" containerID="15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650" exitCode=0
Jul 01 19:35:08.915206 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.914817    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86c5d8b9df-m72x2"
Jul 01 19:35:08.915206 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.914860    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c5d8b9df-m72x2" event={"ID":"0cb2b748-eaf3-4c51-acd8-691594ad984c","Type":"ContainerDied","Data":"15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650"}
Jul 01 19:35:08.915206 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.914916    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86c5d8b9df-m72x2" event={"ID":"0cb2b748-eaf3-4c51-acd8-691594ad984c","Type":"ContainerDied","Data":"fcf0e1cd5c1eba42916f0fbebad4e2cbaf658ca86a5a25d60fb01682d1e4bc53"}
Jul 01 19:35:08.915206 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.914937    2586 scope.go:117] "RemoveContainer" containerID="15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650"
Jul 01 19:35:08.925990 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.925974    2586 scope.go:117] "RemoveContainer" containerID="15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650"
Jul 01 19:35:08.926217 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:35:08.926200    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650\": container with ID starting with 15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650 not found: ID does not exist" containerID="15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650"
Jul 01 19:35:08.926262 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.926222    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650"} err="failed to get container status \"15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650\": rpc error: code = NotFound desc = could not find container \"15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650\": container with ID starting with 15b9eb985704acd169f50106c1b86be152214a21b7e0411bba97fc4514fb0650 not found: ID does not exist"
Jul 01 19:35:08.937905 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.937881    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86c5d8b9df-m72x2"]
Jul 01 19:35:08.942144 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:08.942125    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86c5d8b9df-m72x2"]
Jul 01 19:35:09.883422 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:35:09.883391    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cb2b748-eaf3-4c51-acd8-691594ad984c" path="/var/lib/kubelet/pods/0cb2b748-eaf3-4c51-acd8-691594ad984c/volumes"
Jul 01 19:37:17.018176 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:37:17.018146    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:39:06.170002 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:39:06.169951    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:39:07.376357 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:39:07.376319    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:40:38.129106 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:40:38.129072    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:41:30.024859 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:30.024820    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:41:48.633323 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:48.633284    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"]
Jul 01 19:41:48.633913 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:48.633534    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" podUID="795cf749-d561-441f-ade3-9ad1ae0f5c08" containerName="switch-graph-6c6c1" containerID="cri-o://89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07" gracePeriod=30
Jul 01 19:41:48.659594 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:48.659561    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:41:48.812830 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:48.812800    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:41:48.898610 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:48.898518    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:41:48.979926 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:48.979888    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:41:49.101811 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:49.101785    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:41:49.230261 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:49.230232    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:41:49.358465 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:49.358432    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:41:51.591251 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:51.591206    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" podUID="795cf749-d561-441f-ade3-9ad1ae0f5c08" containerName="switch-graph-6c6c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:41:56.591141 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:41:56.591099    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" podUID="795cf749-d561-441f-ade3-9ad1ae0f5c08" containerName="switch-graph-6c6c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:42:01.591943 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:01.591902    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" podUID="795cf749-d561-441f-ade3-9ad1ae0f5c08" containerName="switch-graph-6c6c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:42:01.592490 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:01.592019    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:42:06.590881 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:06.590840    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" podUID="795cf749-d561-441f-ade3-9ad1ae0f5c08" containerName="switch-graph-6c6c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:42:11.591780 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:11.591743    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" podUID="795cf749-d561-441f-ade3-9ad1ae0f5c08" containerName="switch-graph-6c6c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:42:16.590913 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:16.590872    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" podUID="795cf749-d561-441f-ade3-9ad1ae0f5c08" containerName="switch-graph-6c6c1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:42:18.773718 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:18.773677    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:42:18.811093 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:18.811060    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795cf749-d561-441f-ade3-9ad1ae0f5c08-openshift-service-ca-bundle\") pod \"795cf749-d561-441f-ade3-9ad1ae0f5c08\" (UID: \"795cf749-d561-441f-ade3-9ad1ae0f5c08\") "
Jul 01 19:42:18.811093 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:18.811098    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795cf749-d561-441f-ade3-9ad1ae0f5c08-proxy-tls\") pod \"795cf749-d561-441f-ade3-9ad1ae0f5c08\" (UID: \"795cf749-d561-441f-ade3-9ad1ae0f5c08\") "
Jul 01 19:42:18.811422 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:18.811398    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795cf749-d561-441f-ade3-9ad1ae0f5c08-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "795cf749-d561-441f-ade3-9ad1ae0f5c08" (UID: "795cf749-d561-441f-ade3-9ad1ae0f5c08"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:18.813090 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:18.813064    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795cf749-d561-441f-ade3-9ad1ae0f5c08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "795cf749-d561-441f-ade3-9ad1ae0f5c08" (UID: "795cf749-d561-441f-ade3-9ad1ae0f5c08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:42:18.912339 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:18.912255    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795cf749-d561-441f-ade3-9ad1ae0f5c08-openshift-service-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:42:18.912339 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:18.912282    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795cf749-d561-441f-ade3-9ad1ae0f5c08-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:42:19.375216 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:19.375178    2586 generic.go:356] "Generic (PLEG): container finished" podID="795cf749-d561-441f-ade3-9ad1ae0f5c08" containerID="89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07" exitCode=0
Jul 01 19:42:19.375216 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:19.375217    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" event={"ID":"795cf749-d561-441f-ade3-9ad1ae0f5c08","Type":"ContainerDied","Data":"89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07"}
Jul 01 19:42:19.375441 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:19.375237    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"
Jul 01 19:42:19.375441 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:19.375280    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn" event={"ID":"795cf749-d561-441f-ade3-9ad1ae0f5c08","Type":"ContainerDied","Data":"531fdd91a79666105ed2de52cf88fa5617544df1c908fe99bc4431d66cdd00db"}
Jul 01 19:42:19.375441 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:19.375302    2586 scope.go:117] "RemoveContainer" containerID="89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07"
Jul 01 19:42:19.384571 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:19.384558    2586 scope.go:117] "RemoveContainer" containerID="89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07"
Jul 01 19:42:19.384832 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:42:19.384814    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07\": container with ID starting with 89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07 not found: ID does not exist" containerID="89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07"
Jul 01 19:42:19.384896 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:19.384840    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07"} err="failed to get container status \"89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07\": rpc error: code = NotFound desc = could not find container \"89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07\": container with ID starting with 89af1a439becceb8c372ffe92f39b0c5cd40409308fae1a2b768b23a4989fe07 not found: ID does not exist"
Jul 01 19:42:19.400371 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:19.400338    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"]
Jul 01 19:42:19.403578 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:19.403556    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c6c1-578d4456d5-mfhwn"]
Jul 01 19:42:19.882993 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:19.882958    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795cf749-d561-441f-ade3-9ad1ae0f5c08" path="/var/lib/kubelet/pods/795cf749-d561-441f-ade3-9ad1ae0f5c08/volumes"
Jul 01 19:42:28.652984 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:28.652954    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"]
Jul 01 19:42:28.653375 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:28.653175    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" podUID="0eed7b11-4b52-4838-a9b3-103c34b46928" containerName="sequence-graph-d109c" containerID="cri-o://6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab" gracePeriod=30
Jul 01 19:42:28.675780 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:28.675749    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:28.777867 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:28.777830    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:28.852414 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:28.852386    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:28.933184 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:28.933107    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:29.330704 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:29.330646    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:31.730929 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:31.730886    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" podUID="0eed7b11-4b52-4838-a9b3-103c34b46928" containerName="sequence-graph-d109c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:42:36.731202 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:36.731163    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" podUID="0eed7b11-4b52-4838-a9b3-103c34b46928" containerName="sequence-graph-d109c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:42:41.731583 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:41.731539    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" podUID="0eed7b11-4b52-4838-a9b3-103c34b46928" containerName="sequence-graph-d109c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:42:41.731977 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:41.731647    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:42:46.731616 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:46.731575    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" podUID="0eed7b11-4b52-4838-a9b3-103c34b46928" containerName="sequence-graph-d109c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:42:48.851083 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.851049    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"]
Jul 01 19:42:48.851460 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.851298    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795cf749-d561-441f-ade3-9ad1ae0f5c08" containerName="switch-graph-6c6c1"
Jul 01 19:42:48.851460 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.851309    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="795cf749-d561-441f-ade3-9ad1ae0f5c08" containerName="switch-graph-6c6c1"
Jul 01 19:42:48.851460 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.851322    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb2b748-eaf3-4c51-acd8-691594ad984c" containerName="console"
Jul 01 19:42:48.851460 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.851328    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb2b748-eaf3-4c51-acd8-691594ad984c" containerName="console"
Jul 01 19:42:48.851460 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.851373    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb2b748-eaf3-4c51-acd8-691594ad984c" containerName="console"
Jul 01 19:42:48.851460 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.851381    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="795cf749-d561-441f-ade3-9ad1ae0f5c08" containerName="switch-graph-6c6c1"
Jul 01 19:42:48.855494 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.855472    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:42:48.857765 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.857747    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-44b6c-kube-rbac-proxy-sar-config\""
Jul 01 19:42:48.857846 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.857756    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-44b6c-serving-cert\""
Jul 01 19:42:48.866135 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.866109    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:48.866287 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.866253    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"]
Jul 01 19:42:48.937989 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.937961    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7fe6464-7430-43d8-9b79-b5933d2cd88b-openshift-service-ca-bundle\") pod \"ensemble-graph-44b6c-867f7d49d8-w2qst\" (UID: \"a7fe6464-7430-43d8-9b79-b5933d2cd88b\") " pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:42:48.938146 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:48.937999    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7fe6464-7430-43d8-9b79-b5933d2cd88b-proxy-tls\") pod \"ensemble-graph-44b6c-867f7d49d8-w2qst\" (UID: \"a7fe6464-7430-43d8-9b79-b5933d2cd88b\") " pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:42:49.038834 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:49.038804    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7fe6464-7430-43d8-9b79-b5933d2cd88b-openshift-service-ca-bundle\") pod \"ensemble-graph-44b6c-867f7d49d8-w2qst\" (UID: \"a7fe6464-7430-43d8-9b79-b5933d2cd88b\") " pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:42:49.038997 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:49.038842    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7fe6464-7430-43d8-9b79-b5933d2cd88b-proxy-tls\") pod \"ensemble-graph-44b6c-867f7d49d8-w2qst\" (UID: \"a7fe6464-7430-43d8-9b79-b5933d2cd88b\") " pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:42:49.039516 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:49.039492    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7fe6464-7430-43d8-9b79-b5933d2cd88b-openshift-service-ca-bundle\") pod \"ensemble-graph-44b6c-867f7d49d8-w2qst\" (UID: \"a7fe6464-7430-43d8-9b79-b5933d2cd88b\") " pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:42:49.041234 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:49.041212    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7fe6464-7430-43d8-9b79-b5933d2cd88b-proxy-tls\") pod \"ensemble-graph-44b6c-867f7d49d8-w2qst\" (UID: \"a7fe6464-7430-43d8-9b79-b5933d2cd88b\") " pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:42:49.096060 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:49.096030    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:49.164535 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:49.164461    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:42:49.288598 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:49.288565    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"]
Jul 01 19:42:49.291589 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:42:49.291565    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fe6464_7430_43d8_9b79_b5933d2cd88b.slice/crio-ae7a8c876f46111b878979a637de70260801d10f6988b6a8d9c6b53d8fdbb1a9 WatchSource:0}: Error finding container ae7a8c876f46111b878979a637de70260801d10f6988b6a8d9c6b53d8fdbb1a9: Status 404 returned error can't find the container with id ae7a8c876f46111b878979a637de70260801d10f6988b6a8d9c6b53d8fdbb1a9
Jul 01 19:42:49.484089 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:49.484058    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" event={"ID":"a7fe6464-7430-43d8-9b79-b5933d2cd88b","Type":"ContainerStarted","Data":"81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb"}
Jul 01 19:42:49.484279 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:49.484108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" event={"ID":"a7fe6464-7430-43d8-9b79-b5933d2cd88b","Type":"ContainerStarted","Data":"ae7a8c876f46111b878979a637de70260801d10f6988b6a8d9c6b53d8fdbb1a9"}
Jul 01 19:42:49.484279 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:49.484156    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:42:49.500943 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:49.500891    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" podStartSLOduration=1.500878392 podStartE2EDuration="1.500878392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:42:49.500056467 +0000 UTC m=+1284.046305807" watchObservedRunningTime="2026-07-01 19:42:49.500878392 +0000 UTC m=+1284.047127727"
Jul 01 19:42:51.731545 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:51.731507    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" podUID="0eed7b11-4b52-4838-a9b3-103c34b46928" containerName="sequence-graph-d109c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:42:52.617646 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:52.617604    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:55.493258 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:55.493229    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:42:56.732047 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:56.732001    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" podUID="0eed7b11-4b52-4838-a9b3-103c34b46928" containerName="sequence-graph-d109c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:42:58.794294 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:58.794272    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:42:58.911413 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:58.911323    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eed7b11-4b52-4838-a9b3-103c34b46928-proxy-tls\") pod \"0eed7b11-4b52-4838-a9b3-103c34b46928\" (UID: \"0eed7b11-4b52-4838-a9b3-103c34b46928\") "
Jul 01 19:42:58.911413 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:58.911395    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eed7b11-4b52-4838-a9b3-103c34b46928-openshift-service-ca-bundle\") pod \"0eed7b11-4b52-4838-a9b3-103c34b46928\" (UID: \"0eed7b11-4b52-4838-a9b3-103c34b46928\") "
Jul 01 19:42:58.911946 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:58.911912    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eed7b11-4b52-4838-a9b3-103c34b46928-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0eed7b11-4b52-4838-a9b3-103c34b46928" (UID: "0eed7b11-4b52-4838-a9b3-103c34b46928"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:58.914036 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:58.914012    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eed7b11-4b52-4838-a9b3-103c34b46928-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0eed7b11-4b52-4838-a9b3-103c34b46928" (UID: "0eed7b11-4b52-4838-a9b3-103c34b46928"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:42:58.923211 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:58.923187    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"]
Jul 01 19:42:58.923409 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:58.923386    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" podUID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" containerName="ensemble-graph-44b6c" containerID="cri-o://81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb" gracePeriod=30
Jul 01 19:42:58.955356 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:58.955316    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:59.012654 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.012624    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eed7b11-4b52-4838-a9b3-103c34b46928-openshift-service-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:42:59.012654 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.012648    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0eed7b11-4b52-4838-a9b3-103c34b46928-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:42:59.090105 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.090074    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:59.185966 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.185874    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:59.260438 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.260407    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:59.369781 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.369749    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:42:59.518288 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.518257    2586 generic.go:356] "Generic (PLEG): container finished" podID="0eed7b11-4b52-4838-a9b3-103c34b46928" containerID="6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab" exitCode=0
Jul 01 19:42:59.518462 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.518300    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"
Jul 01 19:42:59.518462 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.518311    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" event={"ID":"0eed7b11-4b52-4838-a9b3-103c34b46928","Type":"ContainerDied","Data":"6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab"}
Jul 01 19:42:59.518462 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.518353    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr" event={"ID":"0eed7b11-4b52-4838-a9b3-103c34b46928","Type":"ContainerDied","Data":"8c4642254dedd8a11791bc7c71628c790fecf0da678b2b54bbf0f87b73c3c8a1"}
Jul 01 19:42:59.518462 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.518375    2586 scope.go:117] "RemoveContainer" containerID="6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab"
Jul 01 19:42:59.526663 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.526646    2586 scope.go:117] "RemoveContainer" containerID="6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab"
Jul 01 19:42:59.526941 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:42:59.526924    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab\": container with ID starting with 6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab not found: ID does not exist" containerID="6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab"
Jul 01 19:42:59.527000 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.526951    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab"} err="failed to get container status \"6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab\": rpc error: code = NotFound desc = could not find container \"6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab\": container with ID starting with 6f613654511a06fa6d8838a9b96951df4cdab10adf3b5b86c5c7d25644bf04ab not found: ID does not exist"
Jul 01 19:42:59.538434 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.538410    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"]
Jul 01 19:42:59.542704 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.542668    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-d109c-6ddb5c8b58-56ppr"]
Jul 01 19:42:59.882387 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:42:59.882310    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eed7b11-4b52-4838-a9b3-103c34b46928" path="/var/lib/kubelet/pods/0eed7b11-4b52-4838-a9b3-103c34b46928/volumes"
Jul 01 19:43:00.103703 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:00.103655    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:43:00.491670 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:00.491631    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" podUID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" containerName="ensemble-graph-44b6c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:43:05.492461 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:05.492421    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" podUID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" containerName="ensemble-graph-44b6c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:43:10.491802 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:10.491762    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" podUID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" containerName="ensemble-graph-44b6c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:43:10.492282 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:10.491878    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:43:15.491275 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:15.491235    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" podUID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" containerName="ensemble-graph-44b6c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:43:20.490875 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:20.490829    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" podUID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" containerName="ensemble-graph-44b6c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:43:25.491169 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:25.491128    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" podUID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" containerName="ensemble-graph-44b6c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:43:28.831190 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.831153    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"]
Jul 01 19:43:28.831539 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.831423    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eed7b11-4b52-4838-a9b3-103c34b46928" containerName="sequence-graph-d109c"
Jul 01 19:43:28.831539 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.831435    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eed7b11-4b52-4838-a9b3-103c34b46928" containerName="sequence-graph-d109c"
Jul 01 19:43:28.831539 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.831481    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eed7b11-4b52-4838-a9b3-103c34b46928" containerName="sequence-graph-d109c"
Jul 01 19:43:28.834467 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.834448    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:28.835219 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.835194    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55a335f9-b53e-4de6-a17e-28b5ebb0e737-openshift-service-ca-bundle\") pod \"sequence-graph-f7f91-5c7b545686-vfvqz\" (UID: \"55a335f9-b53e-4de6-a17e-28b5ebb0e737\") " pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:28.835320 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.835241    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55a335f9-b53e-4de6-a17e-28b5ebb0e737-proxy-tls\") pod \"sequence-graph-f7f91-5c7b545686-vfvqz\" (UID: \"55a335f9-b53e-4de6-a17e-28b5ebb0e737\") " pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:28.836967 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.836949    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f7f91-serving-cert\""
Jul 01 19:43:28.837084 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.837016    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f7f91-kube-rbac-proxy-sar-config\""
Jul 01 19:43:28.838223 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.838204    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:43:28.844113 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.844090    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"]
Jul 01 19:43:28.935750 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.935722    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55a335f9-b53e-4de6-a17e-28b5ebb0e737-openshift-service-ca-bundle\") pod \"sequence-graph-f7f91-5c7b545686-vfvqz\" (UID: \"55a335f9-b53e-4de6-a17e-28b5ebb0e737\") " pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:28.935864 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.935809    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55a335f9-b53e-4de6-a17e-28b5ebb0e737-proxy-tls\") pod \"sequence-graph-f7f91-5c7b545686-vfvqz\" (UID: \"55a335f9-b53e-4de6-a17e-28b5ebb0e737\") " pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:28.936009 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:43:28.935946    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-f7f91-serving-cert: secret "sequence-graph-f7f91-serving-cert" not found
Jul 01 19:43:28.936068 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:43:28.936010    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a335f9-b53e-4de6-a17e-28b5ebb0e737-proxy-tls podName:55a335f9-b53e-4de6-a17e-28b5ebb0e737 nodeName:}" failed. No retries permitted until 2026-07-01 19:43:29.435990695 +0000 UTC m=+1323.982240026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/55a335f9-b53e-4de6-a17e-28b5ebb0e737-proxy-tls") pod "sequence-graph-f7f91-5c7b545686-vfvqz" (UID: "55a335f9-b53e-4de6-a17e-28b5ebb0e737") : secret "sequence-graph-f7f91-serving-cert" not found
Jul 01 19:43:28.936628 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:28.936604    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55a335f9-b53e-4de6-a17e-28b5ebb0e737-openshift-service-ca-bundle\") pod \"sequence-graph-f7f91-5c7b545686-vfvqz\" (UID: \"55a335f9-b53e-4de6-a17e-28b5ebb0e737\") " pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:29.070852 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.070825    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:43:29.137278 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.137207    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7fe6464-7430-43d8-9b79-b5933d2cd88b-proxy-tls\") pod \"a7fe6464-7430-43d8-9b79-b5933d2cd88b\" (UID: \"a7fe6464-7430-43d8-9b79-b5933d2cd88b\") "
Jul 01 19:43:29.137410 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.137278    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7fe6464-7430-43d8-9b79-b5933d2cd88b-openshift-service-ca-bundle\") pod \"a7fe6464-7430-43d8-9b79-b5933d2cd88b\" (UID: \"a7fe6464-7430-43d8-9b79-b5933d2cd88b\") "
Jul 01 19:43:29.137629 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.137597    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fe6464-7430-43d8-9b79-b5933d2cd88b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a7fe6464-7430-43d8-9b79-b5933d2cd88b" (UID: "a7fe6464-7430-43d8-9b79-b5933d2cd88b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:43:29.139169 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.139149    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7fe6464-7430-43d8-9b79-b5933d2cd88b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7fe6464-7430-43d8-9b79-b5933d2cd88b" (UID: "a7fe6464-7430-43d8-9b79-b5933d2cd88b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:43:29.237894 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.237862    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7fe6464-7430-43d8-9b79-b5933d2cd88b-openshift-service-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:43:29.237894 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.237891    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7fe6464-7430-43d8-9b79-b5933d2cd88b-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:43:29.249986 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.249958    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:43:29.438811 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.438717    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55a335f9-b53e-4de6-a17e-28b5ebb0e737-proxy-tls\") pod \"sequence-graph-f7f91-5c7b545686-vfvqz\" (UID: \"55a335f9-b53e-4de6-a17e-28b5ebb0e737\") " pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:29.441105 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.441081    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55a335f9-b53e-4de6-a17e-28b5ebb0e737-proxy-tls\") pod \"sequence-graph-f7f91-5c7b545686-vfvqz\" (UID: \"55a335f9-b53e-4de6-a17e-28b5ebb0e737\") " pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:29.444971 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.444955    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:29.567167 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.567137    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"]
Jul 01 19:43:29.569915 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:43:29.569890    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a335f9_b53e_4de6_a17e_28b5ebb0e737.slice/crio-24d777b3de1507a72b771294403e1eda1a885e47c0930db4a950a38190a848e9 WatchSource:0}: Error finding container 24d777b3de1507a72b771294403e1eda1a885e47c0930db4a950a38190a848e9: Status 404 returned error can't find the container with id 24d777b3de1507a72b771294403e1eda1a885e47c0930db4a950a38190a848e9
Jul 01 19:43:29.624745 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.624713    2586 generic.go:356] "Generic (PLEG): container finished" podID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" containerID="81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb" exitCode=0
Jul 01 19:43:29.624847 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.624760    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"
Jul 01 19:43:29.624847 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.624801    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" event={"ID":"a7fe6464-7430-43d8-9b79-b5933d2cd88b","Type":"ContainerDied","Data":"81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb"}
Jul 01 19:43:29.624939 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.624863    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst" event={"ID":"a7fe6464-7430-43d8-9b79-b5933d2cd88b","Type":"ContainerDied","Data":"ae7a8c876f46111b878979a637de70260801d10f6988b6a8d9c6b53d8fdbb1a9"}
Jul 01 19:43:29.624939 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.624897    2586 scope.go:117] "RemoveContainer" containerID="81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb"
Jul 01 19:43:29.626556 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.626536    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" event={"ID":"55a335f9-b53e-4de6-a17e-28b5ebb0e737","Type":"ContainerStarted","Data":"24d777b3de1507a72b771294403e1eda1a885e47c0930db4a950a38190a848e9"}
Jul 01 19:43:29.633268 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.633250    2586 scope.go:117] "RemoveContainer" containerID="81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb"
Jul 01 19:43:29.633539 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:43:29.633519    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb\": container with ID starting with 81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb not found: ID does not exist" containerID="81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb"
Jul 01 19:43:29.633600 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.633547    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb"} err="failed to get container status \"81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb\": rpc error: code = NotFound desc = could not find container \"81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb\": container with ID starting with 81a9492bf22e77d6ef2ba843d67af4f4c6ff15aa2888d753bf7393ade0ef06cb not found: ID does not exist"
Jul 01 19:43:29.645590 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.645565    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"]
Jul 01 19:43:29.649710 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.649670    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-44b6c-867f7d49d8-w2qst"]
Jul 01 19:43:29.882888 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:29.882847    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" path="/var/lib/kubelet/pods/a7fe6464-7430-43d8-9b79-b5933d2cd88b/volumes"
Jul 01 19:43:30.633004 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:30.632971    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" event={"ID":"55a335f9-b53e-4de6-a17e-28b5ebb0e737","Type":"ContainerStarted","Data":"3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4"}
Jul 01 19:43:30.633227 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:30.633105    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:30.650987 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:30.650929    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" podStartSLOduration=2.650910042 podStartE2EDuration="2.650910042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:43:30.64981233 +0000 UTC m=+1325.196061666" watchObservedRunningTime="2026-07-01 19:43:30.650910042 +0000 UTC m=+1325.197159376"
Jul 01 19:43:36.642516 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:36.642474    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:38.936169 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:38.936130    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"]
Jul 01 19:43:38.936625 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:38.936378    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" podUID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" containerName="sequence-graph-f7f91" containerID="cri-o://3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4" gracePeriod=30
Jul 01 19:43:38.962641 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:38.962608    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:43:39.080625 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:39.080594    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:43:39.183592 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:39.183559    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:43:39.280940 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:39.280903    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:43:39.380379 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:39.380341    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:43:39.481007 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:39.480975    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:43:41.640217 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:41.640176    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" podUID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" containerName="sequence-graph-f7f91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:43:46.640214 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:46.640168    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" podUID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" containerName="sequence-graph-f7f91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:43:51.640410 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:51.640327    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" podUID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" containerName="sequence-graph-f7f91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:43:51.640870 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:51.640431    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:43:56.640211 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:43:56.640175    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" podUID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" containerName="sequence-graph-f7f91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:44:01.640319 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:01.640274    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" podUID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" containerName="sequence-graph-f7f91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:44:06.640244 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:06.640199    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" podUID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" containerName="sequence-graph-f7f91" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:44:09.176076 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.176043    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"]
Jul 01 19:44:09.176414 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.176315    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" containerName="ensemble-graph-44b6c"
Jul 01 19:44:09.176414 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.176325    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" containerName="ensemble-graph-44b6c"
Jul 01 19:44:09.176414 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.176376    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7fe6464-7430-43d8-9b79-b5933d2cd88b" containerName="ensemble-graph-44b6c"
Jul 01 19:44:09.179128 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.179109    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:44:09.181495 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.181472    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-25c73-serving-cert\""
Jul 01 19:44:09.181495 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.181487    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-25c73-kube-rbac-proxy-sar-config\""
Jul 01 19:44:09.187712 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.187669    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"]
Jul 01 19:44:09.194049 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.193974    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:44:09.244914 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.244876    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a34d86f-d248-48de-b206-3a031a627764-proxy-tls\") pod \"ensemble-graph-25c73-765966cfb4-d8pwg\" (UID: \"3a34d86f-d248-48de-b206-3a031a627764\") " pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:44:09.245063 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.244935    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a34d86f-d248-48de-b206-3a031a627764-openshift-service-ca-bundle\") pod \"ensemble-graph-25c73-765966cfb4-d8pwg\" (UID: \"3a34d86f-d248-48de-b206-3a031a627764\") " pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:44:09.345888 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.345849    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a34d86f-d248-48de-b206-3a031a627764-openshift-service-ca-bundle\") pod \"ensemble-graph-25c73-765966cfb4-d8pwg\" (UID: \"3a34d86f-d248-48de-b206-3a031a627764\") " pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:44:09.346081 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.345919    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a34d86f-d248-48de-b206-3a031a627764-proxy-tls\") pod \"ensemble-graph-25c73-765966cfb4-d8pwg\" (UID: \"3a34d86f-d248-48de-b206-3a031a627764\") " pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:44:09.346589 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.346564    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a34d86f-d248-48de-b206-3a031a627764-openshift-service-ca-bundle\") pod \"ensemble-graph-25c73-765966cfb4-d8pwg\" (UID: \"3a34d86f-d248-48de-b206-3a031a627764\") " pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:44:09.348222 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.348204    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a34d86f-d248-48de-b206-3a031a627764-proxy-tls\") pod \"ensemble-graph-25c73-765966cfb4-d8pwg\" (UID: \"3a34d86f-d248-48de-b206-3a031a627764\") " pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:44:09.489984 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.489955    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:44:09.587600 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.587574    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:44:09.630033 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.630003    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"]
Jul 01 19:44:09.633268 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:44:09.633237    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a34d86f_d248_48de_b206_3a031a627764.slice/crio-4ea05f83b1cdf4757b12fad0bdb9d38eb8f9bc5cb7d7f3cbf22eb2a46f4186a4 WatchSource:0}: Error finding container 4ea05f83b1cdf4757b12fad0bdb9d38eb8f9bc5cb7d7f3cbf22eb2a46f4186a4: Status 404 returned error can't find the container with id 4ea05f83b1cdf4757b12fad0bdb9d38eb8f9bc5cb7d7f3cbf22eb2a46f4186a4
Jul 01 19:44:09.648759 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.648740    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55a335f9-b53e-4de6-a17e-28b5ebb0e737-proxy-tls\") pod \"55a335f9-b53e-4de6-a17e-28b5ebb0e737\" (UID: \"55a335f9-b53e-4de6-a17e-28b5ebb0e737\") "
Jul 01 19:44:09.648875 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.648779    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55a335f9-b53e-4de6-a17e-28b5ebb0e737-openshift-service-ca-bundle\") pod \"55a335f9-b53e-4de6-a17e-28b5ebb0e737\" (UID: \"55a335f9-b53e-4de6-a17e-28b5ebb0e737\") "
Jul 01 19:44:09.649144 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.649121    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a335f9-b53e-4de6-a17e-28b5ebb0e737-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "55a335f9-b53e-4de6-a17e-28b5ebb0e737" (UID: "55a335f9-b53e-4de6-a17e-28b5ebb0e737"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:44:09.650567 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.650541    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a335f9-b53e-4de6-a17e-28b5ebb0e737-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55a335f9-b53e-4de6-a17e-28b5ebb0e737" (UID: "55a335f9-b53e-4de6-a17e-28b5ebb0e737"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:44:09.749303 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.749220    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55a335f9-b53e-4de6-a17e-28b5ebb0e737-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:44:09.749303 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.749249    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55a335f9-b53e-4de6-a17e-28b5ebb0e737-openshift-service-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:44:09.767775 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.767745    2586 generic.go:356] "Generic (PLEG): container finished" podID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" containerID="3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4" exitCode=0
Jul 01 19:44:09.767931 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.767789    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"
Jul 01 19:44:09.767931 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.767791    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" event={"ID":"55a335f9-b53e-4de6-a17e-28b5ebb0e737","Type":"ContainerDied","Data":"3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4"}
Jul 01 19:44:09.767931 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.767845    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz" event={"ID":"55a335f9-b53e-4de6-a17e-28b5ebb0e737","Type":"ContainerDied","Data":"24d777b3de1507a72b771294403e1eda1a885e47c0930db4a950a38190a848e9"}
Jul 01 19:44:09.767931 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.767864    2586 scope.go:117] "RemoveContainer" containerID="3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4"
Jul 01 19:44:09.769351 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.769325    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" event={"ID":"3a34d86f-d248-48de-b206-3a031a627764","Type":"ContainerStarted","Data":"d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906"}
Jul 01 19:44:09.769466 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.769376    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" event={"ID":"3a34d86f-d248-48de-b206-3a031a627764","Type":"ContainerStarted","Data":"4ea05f83b1cdf4757b12fad0bdb9d38eb8f9bc5cb7d7f3cbf22eb2a46f4186a4"}
Jul 01 19:44:09.769466 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.769408    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:44:09.776187 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.776171    2586 scope.go:117] "RemoveContainer" containerID="3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4"
Jul 01 19:44:09.776443 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:44:09.776420    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4\": container with ID starting with 3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4 not found: ID does not exist" containerID="3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4"
Jul 01 19:44:09.776538 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.776449    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4"} err="failed to get container status \"3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4\": rpc error: code = NotFound desc = could not find container \"3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4\": container with ID starting with 3547cf7ec85aed5b3db4e93ea8dbfd8cb62993ce448c464564d070eafff4b5d4 not found: ID does not exist"
Jul 01 19:44:09.793512 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.793475    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" podStartSLOduration=0.793461099 podStartE2EDuration="793.461099ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:44:09.793305151 +0000 UTC m=+1364.339554488" watchObservedRunningTime="2026-07-01 19:44:09.793461099 +0000 UTC m=+1364.339710435"
Jul 01 19:44:09.807550 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.807526    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"]
Jul 01 19:44:09.811183 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.811164    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f7f91-5c7b545686-vfvqz"]
Jul 01 19:44:09.883395 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:09.883366    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" path="/var/lib/kubelet/pods/55a335f9-b53e-4de6-a17e-28b5ebb0e737/volumes"
Jul 01 19:44:10.334921 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:10.334890    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:44:15.780301 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:15.780273    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:44:39.168155 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.168074    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"]
Jul 01 19:44:39.168573 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.168407    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" containerName="sequence-graph-f7f91"
Jul 01 19:44:39.168573 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.168422    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" containerName="sequence-graph-f7f91"
Jul 01 19:44:39.168573 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.168481    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="55a335f9-b53e-4de6-a17e-28b5ebb0e737" containerName="sequence-graph-f7f91"
Jul 01 19:44:39.172512 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.172490    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:44:39.174977 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.174952    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f705b-kube-rbac-proxy-sar-config\""
Jul 01 19:44:39.175225 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.175200    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f705b-serving-cert\""
Jul 01 19:44:39.176379 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.176354    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:44:39.187432 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.187412    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"]
Jul 01 19:44:39.273392 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.273360    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-openshift-service-ca-bundle\") pod \"sequence-graph-f705b-5fd9c8dfb8-sxjr4\" (UID: \"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac\") " pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:44:39.273545 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.273423    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-proxy-tls\") pod \"sequence-graph-f705b-5fd9c8dfb8-sxjr4\" (UID: \"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac\") " pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:44:39.374309 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.374269    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-proxy-tls\") pod \"sequence-graph-f705b-5fd9c8dfb8-sxjr4\" (UID: \"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac\") " pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:44:39.374481 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.374342    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-openshift-service-ca-bundle\") pod \"sequence-graph-f705b-5fd9c8dfb8-sxjr4\" (UID: \"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac\") " pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:44:39.374481 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:44:39.374433    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-f705b-serving-cert: secret "sequence-graph-f705b-serving-cert" not found
Jul 01 19:44:39.374557 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:44:39.374520    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-proxy-tls podName:bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac nodeName:}" failed. No retries permitted until 2026-07-01 19:44:39.874498003 +0000 UTC m=+1394.420747330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-proxy-tls") pod "sequence-graph-f705b-5fd9c8dfb8-sxjr4" (UID: "bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac") : secret "sequence-graph-f705b-serving-cert" not found
Jul 01 19:44:39.375045 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.375026    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-openshift-service-ca-bundle\") pod \"sequence-graph-f705b-5fd9c8dfb8-sxjr4\" (UID: \"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac\") " pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:44:39.650480 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.650445    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:44:39.877569 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.877533    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-proxy-tls\") pod \"sequence-graph-f705b-5fd9c8dfb8-sxjr4\" (UID: \"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac\") " pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:44:39.880500 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:39.880465    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-proxy-tls\") pod \"sequence-graph-f705b-5fd9c8dfb8-sxjr4\" (UID: \"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac\") " pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:44:40.083072 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:40.083030    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:44:40.206147 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:40.206120    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"]
Jul 01 19:44:40.208860 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:44:40.208833    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf6e0c0_cb7b_45b3_8dc6_513a58ccf8ac.slice/crio-d6825f487e92e09ebc43a9cd2f8ce7cf17aafd28e48e94504c975babc8ca2383 WatchSource:0}: Error finding container d6825f487e92e09ebc43a9cd2f8ce7cf17aafd28e48e94504c975babc8ca2383: Status 404 returned error can't find the container with id d6825f487e92e09ebc43a9cd2f8ce7cf17aafd28e48e94504c975babc8ca2383
Jul 01 19:44:40.886997 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:40.886212    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" event={"ID":"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac","Type":"ContainerStarted","Data":"c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f"}
Jul 01 19:44:40.886997 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:40.886276    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" event={"ID":"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac","Type":"ContainerStarted","Data":"d6825f487e92e09ebc43a9cd2f8ce7cf17aafd28e48e94504c975babc8ca2383"}
Jul 01 19:44:40.886997 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:40.886999    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:44:40.915555 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:40.915511    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" podStartSLOduration=1.9154981389999999 podStartE2EDuration="1.915498139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:44:40.913618088 +0000 UTC m=+1395.459867424" watchObservedRunningTime="2026-07-01 19:44:40.915498139 +0000 UTC m=+1395.461747475"
Jul 01 19:44:47.897155 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:44:47.897119    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:46:48.471916 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:46:48.471810    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:47:08.612081 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:47:08.612046    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:47:40.645420 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:47:40.645349    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:48:28.530957 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:48:28.530841    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:12.511908 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:12.511872    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:23.820030 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:23.819999    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"]
Jul 01 19:52:23.820446 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:23.820217    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" podUID="3a34d86f-d248-48de-b206-3a031a627764" containerName="ensemble-graph-25c73" containerID="cri-o://d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906" gracePeriod=30
Jul 01 19:52:23.853005 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:23.852969    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:23.979802 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:23.979773    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:24.081283 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:24.081195    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:24.184445 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:24.184408    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:24.282830 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:24.282799    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:24.384141 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:24.384054    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:25.779078 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:25.779035    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" podUID="3a34d86f-d248-48de-b206-3a031a627764" containerName="ensemble-graph-25c73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:52:30.778955 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:30.778911    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" podUID="3a34d86f-d248-48de-b206-3a031a627764" containerName="ensemble-graph-25c73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:52:35.779515 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:35.779476    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" podUID="3a34d86f-d248-48de-b206-3a031a627764" containerName="ensemble-graph-25c73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:52:35.779975 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:35.779638    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:52:40.778448 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:40.778405    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" podUID="3a34d86f-d248-48de-b206-3a031a627764" containerName="ensemble-graph-25c73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:52:45.778355 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:45.778318    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" podUID="3a34d86f-d248-48de-b206-3a031a627764" containerName="ensemble-graph-25c73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:52:50.778253 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:50.778164    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" podUID="3a34d86f-d248-48de-b206-3a031a627764" containerName="ensemble-graph-25c73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:52:53.877365 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:53.877329    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"]
Jul 01 19:52:53.877836 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:53.877620    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" podUID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" containerName="sequence-graph-f705b" containerID="cri-o://c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f" gracePeriod=30
Jul 01 19:52:53.897677 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:53.897641    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:53.996068 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:53.996035    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:54.075597 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.075509    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:54.143668 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.143631    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:54.244233 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.244197    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:54.457254 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.457232    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:52:54.490750 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.490725    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a34d86f-d248-48de-b206-3a031a627764-proxy-tls\") pod \"3a34d86f-d248-48de-b206-3a031a627764\" (UID: \"3a34d86f-d248-48de-b206-3a031a627764\") "
Jul 01 19:52:54.490889 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.490758    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a34d86f-d248-48de-b206-3a031a627764-openshift-service-ca-bundle\") pod \"3a34d86f-d248-48de-b206-3a031a627764\" (UID: \"3a34d86f-d248-48de-b206-3a031a627764\") "
Jul 01 19:52:54.491156 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.491125    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a34d86f-d248-48de-b206-3a031a627764-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3a34d86f-d248-48de-b206-3a031a627764" (UID: "3a34d86f-d248-48de-b206-3a031a627764"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:52:54.492729 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.492705    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a34d86f-d248-48de-b206-3a031a627764-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a34d86f-d248-48de-b206-3a031a627764" (UID: "3a34d86f-d248-48de-b206-3a031a627764"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:52:54.542697 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.542642    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:52:54.551807 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.551783    2586 generic.go:356] "Generic (PLEG): container finished" podID="3a34d86f-d248-48de-b206-3a031a627764" containerID="d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906" exitCode=0
Jul 01 19:52:54.551904 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.551834    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"
Jul 01 19:52:54.551904 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.551848    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" event={"ID":"3a34d86f-d248-48de-b206-3a031a627764","Type":"ContainerDied","Data":"d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906"}
Jul 01 19:52:54.551989 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.551904    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg" event={"ID":"3a34d86f-d248-48de-b206-3a031a627764","Type":"ContainerDied","Data":"4ea05f83b1cdf4757b12fad0bdb9d38eb8f9bc5cb7d7f3cbf22eb2a46f4186a4"}
Jul 01 19:52:54.551989 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.551927    2586 scope.go:117] "RemoveContainer" containerID="d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906"
Jul 01 19:52:54.564497 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.564478    2586 scope.go:117] "RemoveContainer" containerID="d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906"
Jul 01 19:52:54.564863 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:52:54.564844    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906\": container with ID starting with d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906 not found: ID does not exist" containerID="d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906"
Jul 01 19:52:54.564926 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.564871    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906"} err="failed to get container status \"d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906\": rpc error: code = NotFound desc = could not find container \"d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906\": container with ID starting with d4838e335493dfb6c9879e6e4f4c81631b449e088e7c0db2085a7e184bcb6906 not found: ID does not exist"
Jul 01 19:52:54.573725 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.573702    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"]
Jul 01 19:52:54.577622 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.577602    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-25c73-765966cfb4-d8pwg"]
Jul 01 19:52:54.591986 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.591942    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a34d86f-d248-48de-b206-3a031a627764-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:52:54.591986 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:54.591961    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a34d86f-d248-48de-b206-3a031a627764-openshift-service-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:52:55.882256 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:55.882226    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a34d86f-d248-48de-b206-3a031a627764" path="/var/lib/kubelet/pods/3a34d86f-d248-48de-b206-3a031a627764/volumes"
Jul 01 19:52:57.896475 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:52:57.896437    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" podUID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" containerName="sequence-graph-f705b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:53:02.896614 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:02.896574    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" podUID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" containerName="sequence-graph-f705b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:53:07.896111 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:07.896064    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" podUID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" containerName="sequence-graph-f705b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:53:07.896499 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:07.896172    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:53:12.895937 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:12.895892    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" podUID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" containerName="sequence-graph-f705b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:53:17.895978 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:17.895940    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" podUID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" containerName="sequence-graph-f705b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:53:22.896183 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:22.896135    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" podUID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" containerName="sequence-graph-f705b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:53:24.023759 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.023734    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:53:24.053884 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.053858    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"]
Jul 01 19:53:24.054137 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.054125    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" containerName="sequence-graph-f705b"
Jul 01 19:53:24.054180 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.054139    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" containerName="sequence-graph-f705b"
Jul 01 19:53:24.054180 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.054153    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a34d86f-d248-48de-b206-3a031a627764" containerName="ensemble-graph-25c73"
Jul 01 19:53:24.054180 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.054158    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a34d86f-d248-48de-b206-3a031a627764" containerName="ensemble-graph-25c73"
Jul 01 19:53:24.054270 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.054202    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a34d86f-d248-48de-b206-3a031a627764" containerName="ensemble-graph-25c73"
Jul 01 19:53:24.054270 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.054211    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" containerName="sequence-graph-f705b"
Jul 01 19:53:24.056970 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.056947    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:53:24.058972 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.058953    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-8003b-kube-rbac-proxy-sar-config\""
Jul 01 19:53:24.059118 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.059102    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-8003b-serving-cert\""
Jul 01 19:53:24.067894 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.067864    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"]
Jul 01 19:53:24.068179 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.068154    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:53:24.116318 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.116291    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-openshift-service-ca-bundle\") pod \"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac\" (UID: \"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac\") "
Jul 01 19:53:24.116484 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.116351    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-proxy-tls\") pod \"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac\" (UID: \"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac\") "
Jul 01 19:53:24.116484 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.116452    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf623a-d9ab-4322-be2d-605f666d3ac7-openshift-service-ca-bundle\") pod \"splitter-graph-8003b-7dc6f58dd9-9kkzl\" (UID: \"7ebf623a-d9ab-4322-be2d-605f666d3ac7\") " pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:53:24.116484 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.116479    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebf623a-d9ab-4322-be2d-605f666d3ac7-proxy-tls\") pod \"splitter-graph-8003b-7dc6f58dd9-9kkzl\" (UID: \"7ebf623a-d9ab-4322-be2d-605f666d3ac7\") " pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:53:24.116708 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.116672    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" (UID: "bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:53:24.118409 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.118390    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" (UID: "bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:53:24.160600 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.160513    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:53:24.217112 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.217080    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf623a-d9ab-4322-be2d-605f666d3ac7-openshift-service-ca-bundle\") pod \"splitter-graph-8003b-7dc6f58dd9-9kkzl\" (UID: \"7ebf623a-d9ab-4322-be2d-605f666d3ac7\") " pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:53:24.217112 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.217114    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebf623a-d9ab-4322-be2d-605f666d3ac7-proxy-tls\") pod \"splitter-graph-8003b-7dc6f58dd9-9kkzl\" (UID: \"7ebf623a-d9ab-4322-be2d-605f666d3ac7\") " pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:53:24.217357 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.217190    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-openshift-service-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:53:24.217357 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.217203    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:53:24.217786 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.217762    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf623a-d9ab-4322-be2d-605f666d3ac7-openshift-service-ca-bundle\") pod \"splitter-graph-8003b-7dc6f58dd9-9kkzl\" (UID: \"7ebf623a-d9ab-4322-be2d-605f666d3ac7\") " pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:53:24.219354 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.219338    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebf623a-d9ab-4322-be2d-605f666d3ac7-proxy-tls\") pod \"splitter-graph-8003b-7dc6f58dd9-9kkzl\" (UID: \"7ebf623a-d9ab-4322-be2d-605f666d3ac7\") " pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:53:24.367492 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.367449    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:53:24.491577 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.491539    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"]
Jul 01 19:53:24.494307 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:53:24.494279    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebf623a_d9ab_4322_be2d_605f666d3ac7.slice/crio-2381536490bd6b8100a9a221a48b63305b9a468e6e66d93d2eabede312d3411b WatchSource:0}: Error finding container 2381536490bd6b8100a9a221a48b63305b9a468e6e66d93d2eabede312d3411b: Status 404 returned error can't find the container with id 2381536490bd6b8100a9a221a48b63305b9a468e6e66d93d2eabede312d3411b
Jul 01 19:53:24.662544 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.662508    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" event={"ID":"7ebf623a-d9ab-4322-be2d-605f666d3ac7","Type":"ContainerStarted","Data":"6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71"}
Jul 01 19:53:24.662803 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.662557    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" event={"ID":"7ebf623a-d9ab-4322-be2d-605f666d3ac7","Type":"ContainerStarted","Data":"2381536490bd6b8100a9a221a48b63305b9a468e6e66d93d2eabede312d3411b"}
Jul 01 19:53:24.662803 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.662592    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:53:24.663759 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.663730    2586 generic.go:356] "Generic (PLEG): container finished" podID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" containerID="c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f" exitCode=0
Jul 01 19:53:24.663879 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.663768    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"
Jul 01 19:53:24.663879 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.663805    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" event={"ID":"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac","Type":"ContainerDied","Data":"c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f"}
Jul 01 19:53:24.663879 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.663847    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4" event={"ID":"bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac","Type":"ContainerDied","Data":"d6825f487e92e09ebc43a9cd2f8ce7cf17aafd28e48e94504c975babc8ca2383"}
Jul 01 19:53:24.663879 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.663875    2586 scope.go:117] "RemoveContainer" containerID="c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f"
Jul 01 19:53:24.672396 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.672380    2586 scope.go:117] "RemoveContainer" containerID="c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f"
Jul 01 19:53:24.672633 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:53:24.672615    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f\": container with ID starting with c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f not found: ID does not exist" containerID="c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f"
Jul 01 19:53:24.672712 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.672641    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f"} err="failed to get container status \"c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f\": rpc error: code = NotFound desc = could not find container \"c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f\": container with ID starting with c17939b482a56052b2ca3bd1d5ccd34bb8a9def27688687f1c3ccef5b2c56a9f not found: ID does not exist"
Jul 01 19:53:24.680221 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.680179    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" podStartSLOduration=0.680165064 podStartE2EDuration="680.165064ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:53:24.680067137 +0000 UTC m=+1919.226316472" watchObservedRunningTime="2026-07-01 19:53:24.680165064 +0000 UTC m=+1919.226414399"
Jul 01 19:53:24.692286 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.692261    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"]
Jul 01 19:53:24.696142 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:24.696106    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f705b-5fd9c8dfb8-sxjr4"]
Jul 01 19:53:25.882852 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:25.882817    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac" path="/var/lib/kubelet/pods/bcf6e0c0-cb7b-45b3-8dc6-513a58ccf8ac/volumes"
Jul 01 19:53:30.674377 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:30.674349    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:53:34.142664 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:34.142591    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"]
Jul 01 19:53:34.143093 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:34.142834    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" podUID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" containerName="splitter-graph-8003b" containerID="cri-o://6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71" gracePeriod=30
Jul 01 19:53:34.165362 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:34.165323    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:53:34.276947 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:34.276907    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:53:34.372593 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:34.372541    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:53:34.448312 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:34.448283    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:53:34.507834 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:34.507793    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:53:34.683265 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:34.683234    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:53:35.672003 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:35.671959    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" podUID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" containerName="splitter-graph-8003b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:53:40.672326 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:40.672282    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" podUID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" containerName="splitter-graph-8003b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:53:45.672446 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:45.672407    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" podUID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" containerName="splitter-graph-8003b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:53:45.672940 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:45.672541    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:53:50.671816 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:50.671779    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" podUID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" containerName="splitter-graph-8003b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:53:54.092252 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.092221    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"]
Jul 01 19:53:54.095429 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.095413    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 19:53:54.097604 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.097584    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-00068-kube-rbac-proxy-sar-config\""
Jul 01 19:53:54.097699 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.097610    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-00068-serving-cert\""
Jul 01 19:53:54.099582 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.099554    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:53:54.105776 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.105757    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"]
Jul 01 19:53:54.245982 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.245948    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad84eab5-b20e-4b2d-829e-11623caf0a1d-openshift-service-ca-bundle\") pod \"switch-graph-00068-5f66c94d7d-sd6wz\" (UID: \"ad84eab5-b20e-4b2d-829e-11623caf0a1d\") " pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 19:53:54.246162 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.245997    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad84eab5-b20e-4b2d-829e-11623caf0a1d-proxy-tls\") pod \"switch-graph-00068-5f66c94d7d-sd6wz\" (UID: \"ad84eab5-b20e-4b2d-829e-11623caf0a1d\") " pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 19:53:54.279458 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.279423    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:53:54.346723 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.346617    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad84eab5-b20e-4b2d-829e-11623caf0a1d-openshift-service-ca-bundle\") pod \"switch-graph-00068-5f66c94d7d-sd6wz\" (UID: \"ad84eab5-b20e-4b2d-829e-11623caf0a1d\") " pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 19:53:54.346723 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.346661    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad84eab5-b20e-4b2d-829e-11623caf0a1d-proxy-tls\") pod \"switch-graph-00068-5f66c94d7d-sd6wz\" (UID: \"ad84eab5-b20e-4b2d-829e-11623caf0a1d\") " pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 19:53:54.347311 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.347285    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad84eab5-b20e-4b2d-829e-11623caf0a1d-openshift-service-ca-bundle\") pod \"switch-graph-00068-5f66c94d7d-sd6wz\" (UID: \"ad84eab5-b20e-4b2d-829e-11623caf0a1d\") " pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 19:53:54.349121 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.349098    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad84eab5-b20e-4b2d-829e-11623caf0a1d-proxy-tls\") pod \"switch-graph-00068-5f66c94d7d-sd6wz\" (UID: \"ad84eab5-b20e-4b2d-829e-11623caf0a1d\") " pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 19:53:54.405975 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.405944    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 19:53:54.533836 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.533802    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"]
Jul 01 19:53:54.536504 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:53:54.536478    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad84eab5_b20e_4b2d_829e_11623caf0a1d.slice/crio-3cf2e9e9c4e15999ecf9b405ac8a526f7d6ce879c0e5e01d4bd9aa94936e54ad WatchSource:0}: Error finding container 3cf2e9e9c4e15999ecf9b405ac8a526f7d6ce879c0e5e01d4bd9aa94936e54ad: Status 404 returned error can't find the container with id 3cf2e9e9c4e15999ecf9b405ac8a526f7d6ce879c0e5e01d4bd9aa94936e54ad
Jul 01 19:53:54.767807 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.767770    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" event={"ID":"ad84eab5-b20e-4b2d-829e-11623caf0a1d","Type":"ContainerStarted","Data":"9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439"}
Jul 01 19:53:54.767972 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.767906    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 19:53:54.767972 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.767949    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" event={"ID":"ad84eab5-b20e-4b2d-829e-11623caf0a1d","Type":"ContainerStarted","Data":"3cf2e9e9c4e15999ecf9b405ac8a526f7d6ce879c0e5e01d4bd9aa94936e54ad"}
Jul 01 19:53:54.784221 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:54.784174    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" podStartSLOduration=0.784159684 podStartE2EDuration="784.159684ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:53:54.782943747 +0000 UTC m=+1949.329193084" watchObservedRunningTime="2026-07-01 19:53:54.784159684 +0000 UTC m=+1949.330409072"
Jul 01 19:53:55.671814 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:53:55.671773    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" podUID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" containerName="splitter-graph-8003b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:54:00.672572 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:00.672533    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" podUID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" containerName="splitter-graph-8003b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:54:00.777831 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:00.777803    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 19:54:04.287999 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.287973    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:54:04.422098 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.422005    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf623a-d9ab-4322-be2d-605f666d3ac7-openshift-service-ca-bundle\") pod \"7ebf623a-d9ab-4322-be2d-605f666d3ac7\" (UID: \"7ebf623a-d9ab-4322-be2d-605f666d3ac7\") "
Jul 01 19:54:04.422098 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.422066    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebf623a-d9ab-4322-be2d-605f666d3ac7-proxy-tls\") pod \"7ebf623a-d9ab-4322-be2d-605f666d3ac7\" (UID: \"7ebf623a-d9ab-4322-be2d-605f666d3ac7\") "
Jul 01 19:54:04.422387 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.422363    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebf623a-d9ab-4322-be2d-605f666d3ac7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7ebf623a-d9ab-4322-be2d-605f666d3ac7" (UID: "7ebf623a-d9ab-4322-be2d-605f666d3ac7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:54:04.424137 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.424112    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebf623a-d9ab-4322-be2d-605f666d3ac7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ebf623a-d9ab-4322-be2d-605f666d3ac7" (UID: "7ebf623a-d9ab-4322-be2d-605f666d3ac7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:54:04.522608 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.522571    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ebf623a-d9ab-4322-be2d-605f666d3ac7-openshift-service-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:54:04.522608 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.522604    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebf623a-d9ab-4322-be2d-605f666d3ac7-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 19:54:04.800036 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.800002    2586 generic.go:356] "Generic (PLEG): container finished" podID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" containerID="6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71" exitCode=0
Jul 01 19:54:04.800235 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.800052    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"
Jul 01 19:54:04.800235 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.800088    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" event={"ID":"7ebf623a-d9ab-4322-be2d-605f666d3ac7","Type":"ContainerDied","Data":"6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71"}
Jul 01 19:54:04.800235 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.800145    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl" event={"ID":"7ebf623a-d9ab-4322-be2d-605f666d3ac7","Type":"ContainerDied","Data":"2381536490bd6b8100a9a221a48b63305b9a468e6e66d93d2eabede312d3411b"}
Jul 01 19:54:04.800235 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.800169    2586 scope.go:117] "RemoveContainer" containerID="6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71"
Jul 01 19:54:04.813566 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.813547    2586 scope.go:117] "RemoveContainer" containerID="6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71"
Jul 01 19:54:04.813861 ip-10-0-141-93 kubenswrapper[2586]: E0701 19:54:04.813838    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71\": container with ID starting with 6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71 not found: ID does not exist" containerID="6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71"
Jul 01 19:54:04.813964 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.813866    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71"} err="failed to get container status \"6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71\": rpc error: code = NotFound desc = could not find container \"6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71\": container with ID starting with 6f8dc34e89e519b10c4cdef906ba7ff8edfbc47611d563e1e08cd802d4835b71 not found: ID does not exist"
Jul 01 19:54:04.822531 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.822506    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"]
Jul 01 19:54:04.824273 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:04.824253    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8003b-7dc6f58dd9-9kkzl"]
Jul 01 19:54:05.883556 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:05.883522    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" path="/var/lib/kubelet/pods/7ebf623a-d9ab-4322-be2d-605f666d3ac7/volumes"
Jul 01 19:54:34.400477 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.400447    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"]
Jul 01 19:54:34.400896 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.400741    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" containerName="splitter-graph-8003b"
Jul 01 19:54:34.400896 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.400754    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" containerName="splitter-graph-8003b"
Jul 01 19:54:34.400896 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.400805    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebf623a-d9ab-4322-be2d-605f666d3ac7" containerName="splitter-graph-8003b"
Jul 01 19:54:34.403736 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.403716    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 19:54:34.406082 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.406059    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5e51f-kube-rbac-proxy-sar-config\""
Jul 01 19:54:34.406199 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.406059    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5e51f-serving-cert\""
Jul 01 19:54:34.416772 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.416749    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"]
Jul 01 19:54:34.426764 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.426736    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6365385e-52e2-4056-8968-2ed2ba96f8ad-proxy-tls\") pod \"splitter-graph-5e51f-6b6dcfc757-955nw\" (UID: \"6365385e-52e2-4056-8968-2ed2ba96f8ad\") " pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 19:54:34.427288 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.427261    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6365385e-52e2-4056-8968-2ed2ba96f8ad-openshift-service-ca-bundle\") pod \"splitter-graph-5e51f-6b6dcfc757-955nw\" (UID: \"6365385e-52e2-4056-8968-2ed2ba96f8ad\") " pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 19:54:34.428125 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.428097    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:54:34.528582 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.528548    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6365385e-52e2-4056-8968-2ed2ba96f8ad-openshift-service-ca-bundle\") pod \"splitter-graph-5e51f-6b6dcfc757-955nw\" (UID: \"6365385e-52e2-4056-8968-2ed2ba96f8ad\") " pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 19:54:34.528795 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.528601    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6365385e-52e2-4056-8968-2ed2ba96f8ad-proxy-tls\") pod \"splitter-graph-5e51f-6b6dcfc757-955nw\" (UID: \"6365385e-52e2-4056-8968-2ed2ba96f8ad\") " pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 19:54:34.529190 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.529166    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6365385e-52e2-4056-8968-2ed2ba96f8ad-openshift-service-ca-bundle\") pod \"splitter-graph-5e51f-6b6dcfc757-955nw\" (UID: \"6365385e-52e2-4056-8968-2ed2ba96f8ad\") " pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 19:54:34.530961 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.530938    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6365385e-52e2-4056-8968-2ed2ba96f8ad-proxy-tls\") pod \"splitter-graph-5e51f-6b6dcfc757-955nw\" (UID: \"6365385e-52e2-4056-8968-2ed2ba96f8ad\") " pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 19:54:34.538645 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.538625    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:54:34.713677 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.713646    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 19:54:34.835103 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.835069    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"]
Jul 01 19:54:34.837931 ip-10-0-141-93 kubenswrapper[2586]: W0701 19:54:34.837907    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6365385e_52e2_4056_8968_2ed2ba96f8ad.slice/crio-903db69fe8e41c2b8ff696c2ba472d594b4178a67381589fef89c3aff659ba52 WatchSource:0}: Error finding container 903db69fe8e41c2b8ff696c2ba472d594b4178a67381589fef89c3aff659ba52: Status 404 returned error can't find the container with id 903db69fe8e41c2b8ff696c2ba472d594b4178a67381589fef89c3aff659ba52
Jul 01 19:54:34.903540 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.903508    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" event={"ID":"6365385e-52e2-4056-8968-2ed2ba96f8ad","Type":"ContainerStarted","Data":"9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481"}
Jul 01 19:54:34.903663 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.903565    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" event={"ID":"6365385e-52e2-4056-8968-2ed2ba96f8ad","Type":"ContainerStarted","Data":"903db69fe8e41c2b8ff696c2ba472d594b4178a67381589fef89c3aff659ba52"}
Jul 01 19:54:34.903663 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.903589    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 19:54:34.919364 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:34.919311    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" podStartSLOduration=0.919295388 podStartE2EDuration="919.295388ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:54:34.918236866 +0000 UTC m=+1989.464486203" watchObservedRunningTime="2026-07-01 19:54:34.919295388 +0000 UTC m=+1989.465544723"
Jul 01 19:54:40.912488 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:54:40.912452    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 19:56:50.284131 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:56:50.284091    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 19:59:08.203784 ip-10-0-141-93 kubenswrapper[2586]: I0701 19:59:08.203750    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:01:04.508051 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:01:04.507987    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:02:05.998541 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:02:05.998502    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:02:36.296505 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:02:36.296466    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:02:49.086201 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:02:49.086157    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"]
Jul 01 20:02:49.086640 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:02:49.086356    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" podUID="6365385e-52e2-4056-8968-2ed2ba96f8ad" containerName="splitter-graph-5e51f" containerID="cri-o://9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481" gracePeriod=30
Jul 01 20:02:49.118964 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:02:49.118933    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:02:49.236906 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:02:49.236871    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:02:49.317887 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:02:49.317851    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:02:50.911014 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:02:50.910973    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" podUID="6365385e-52e2-4056-8968-2ed2ba96f8ad" containerName="splitter-graph-5e51f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:02:55.910475 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:02:55.910430    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" podUID="6365385e-52e2-4056-8968-2ed2ba96f8ad" containerName="splitter-graph-5e51f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:03:00.910511 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:00.910463    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" podUID="6365385e-52e2-4056-8968-2ed2ba96f8ad" containerName="splitter-graph-5e51f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:03:00.910994 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:00.910584    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 20:03:05.910549 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:05.910512    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" podUID="6365385e-52e2-4056-8968-2ed2ba96f8ad" containerName="splitter-graph-5e51f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:03:10.910659 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:10.910611    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" podUID="6365385e-52e2-4056-8968-2ed2ba96f8ad" containerName="splitter-graph-5e51f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:03:15.911318 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:15.911275    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" podUID="6365385e-52e2-4056-8968-2ed2ba96f8ad" containerName="splitter-graph-5e51f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:03:19.230481 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.230452    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 20:03:19.252169 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.252140    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6365385e-52e2-4056-8968-2ed2ba96f8ad-openshift-service-ca-bundle\") pod \"6365385e-52e2-4056-8968-2ed2ba96f8ad\" (UID: \"6365385e-52e2-4056-8968-2ed2ba96f8ad\") "
Jul 01 20:03:19.252328 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.252187    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6365385e-52e2-4056-8968-2ed2ba96f8ad-proxy-tls\") pod \"6365385e-52e2-4056-8968-2ed2ba96f8ad\" (UID: \"6365385e-52e2-4056-8968-2ed2ba96f8ad\") "
Jul 01 20:03:19.252488 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.252463    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6365385e-52e2-4056-8968-2ed2ba96f8ad-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6365385e-52e2-4056-8968-2ed2ba96f8ad" (UID: "6365385e-52e2-4056-8968-2ed2ba96f8ad"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:03:19.254234 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.254211    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6365385e-52e2-4056-8968-2ed2ba96f8ad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6365385e-52e2-4056-8968-2ed2ba96f8ad" (UID: "6365385e-52e2-4056-8968-2ed2ba96f8ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:03:19.352666 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.352575    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6365385e-52e2-4056-8968-2ed2ba96f8ad-openshift-service-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 20:03:19.352666 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.352608    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6365385e-52e2-4056-8968-2ed2ba96f8ad-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 20:03:19.688039 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.687949    2586 generic.go:356] "Generic (PLEG): container finished" podID="6365385e-52e2-4056-8968-2ed2ba96f8ad" containerID="9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481" exitCode=0
Jul 01 20:03:19.688039 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.687996    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"
Jul 01 20:03:19.688272 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.688036    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" event={"ID":"6365385e-52e2-4056-8968-2ed2ba96f8ad","Type":"ContainerDied","Data":"9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481"}
Jul 01 20:03:19.688272 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.688100    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw" event={"ID":"6365385e-52e2-4056-8968-2ed2ba96f8ad","Type":"ContainerDied","Data":"903db69fe8e41c2b8ff696c2ba472d594b4178a67381589fef89c3aff659ba52"}
Jul 01 20:03:19.688272 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.688131    2586 scope.go:117] "RemoveContainer" containerID="9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481"
Jul 01 20:03:19.696336 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.696318    2586 scope.go:117] "RemoveContainer" containerID="9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481"
Jul 01 20:03:19.696585 ip-10-0-141-93 kubenswrapper[2586]: E0701 20:03:19.696567    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481\": container with ID starting with 9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481 not found: ID does not exist" containerID="9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481"
Jul 01 20:03:19.696629 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.696594    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481"} err="failed to get container status \"9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481\": rpc error: code = NotFound desc = could not find container \"9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481\": container with ID starting with 9ee150d7a6774195f3c893a7c6e4ad086187df8fa9c76b528f7e4f431b596481 not found: ID does not exist"
Jul 01 20:03:19.708475 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.708453    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"]
Jul 01 20:03:19.712880 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.712859    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5e51f-6b6dcfc757-955nw"]
Jul 01 20:03:19.883244 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:03:19.883209    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6365385e-52e2-4056-8968-2ed2ba96f8ad" path="/var/lib/kubelet/pods/6365385e-52e2-4056-8968-2ed2ba96f8ad/volumes"
Jul 01 20:05:17.891280 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:05:17.891244    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:07:39.672955 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:07:39.672918    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:10:13.245177 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:13.245143    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"]
Jul 01 20:10:13.245718 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:13.245330    2586 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" podUID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" containerName="switch-graph-00068" containerID="cri-o://9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439" gracePeriod=30
Jul 01 20:10:13.299347 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:13.299310    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:10:13.430428 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:13.430397    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:10:13.492277 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:13.492251    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:10:14.463553 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:14.463522    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:10:14.529190 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:14.529150    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:10:15.779790 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:15.779751    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" podUID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" containerName="switch-graph-00068" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:10:20.774892 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:20.774855    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" podUID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" containerName="switch-graph-00068" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:10:25.775494 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:25.775456    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" podUID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" containerName="switch-graph-00068" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:10:25.776051 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:25.775578    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 20:10:30.167211 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:30.167180    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:30.775512 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:30.775465    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" podUID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" containerName="switch-graph-00068" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:10:31.058970 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:31.058887    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:31.871855 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:31.871827    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:32.661140 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:32.661107    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:33.455206 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:33.455174    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:34.252365 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:34.252336    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:35.054347 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:35.054308    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:35.775203 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:35.775157    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" podUID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" containerName="switch-graph-00068" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:10:35.863483 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:35.863449    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:36.693717 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:36.693675    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:37.508364 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:37.508333    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:38.358175 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:38.358147    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:39.217483 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:39.217454    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:40.017899 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:40.017869    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:40.775500 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:40.775466    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" podUID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" containerName="switch-graph-00068" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 20:10:40.815603 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:40.815578    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-00068-5f66c94d7d-sd6wz_ad84eab5-b20e-4b2d-829e-11623caf0a1d/switch-graph-00068/0.log"
Jul 01 20:10:42.969064 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:42.969033    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:10:43.122348 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.122314    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2sjbp/must-gather-7wtzn"]
Jul 01 20:10:43.122697 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.122663    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6365385e-52e2-4056-8968-2ed2ba96f8ad" containerName="splitter-graph-5e51f"
Jul 01 20:10:43.122697 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.122695    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6365385e-52e2-4056-8968-2ed2ba96f8ad" containerName="splitter-graph-5e51f"
Jul 01 20:10:43.122870 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.122767    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6365385e-52e2-4056-8968-2ed2ba96f8ad" containerName="splitter-graph-5e51f"
Jul 01 20:10:43.125581 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.125563    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2sjbp/must-gather-7wtzn"
Jul 01 20:10:43.129202 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.129179    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2sjbp\"/\"default-dockercfg-m98cx\""
Jul 01 20:10:43.129978 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.129950    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2sjbp\"/\"openshift-service-ca.crt\""
Jul 01 20:10:43.129978 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.129954    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2sjbp\"/\"kube-root-ca.crt\""
Jul 01 20:10:43.138885 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.138863    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2sjbp/must-gather-7wtzn"]
Jul 01 20:10:43.169727 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.169700    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/49c23611-783c-4a3f-86d4-46a3617f78f5-must-gather-output\") pod \"must-gather-7wtzn\" (UID: \"49c23611-783c-4a3f-86d4-46a3617f78f5\") " pod="openshift-must-gather-2sjbp/must-gather-7wtzn"
Jul 01 20:10:43.169860 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.169751    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwst\" (UniqueName: \"kubernetes.io/projected/49c23611-783c-4a3f-86d4-46a3617f78f5-kube-api-access-dfwst\") pod \"must-gather-7wtzn\" (UID: \"49c23611-783c-4a3f-86d4-46a3617f78f5\") " pod="openshift-must-gather-2sjbp/must-gather-7wtzn"
Jul 01 20:10:43.271110 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.271083    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/49c23611-783c-4a3f-86d4-46a3617f78f5-must-gather-output\") pod \"must-gather-7wtzn\" (UID: \"49c23611-783c-4a3f-86d4-46a3617f78f5\") " pod="openshift-must-gather-2sjbp/must-gather-7wtzn"
Jul 01 20:10:43.271230 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.271132    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwst\" (UniqueName: \"kubernetes.io/projected/49c23611-783c-4a3f-86d4-46a3617f78f5-kube-api-access-dfwst\") pod \"must-gather-7wtzn\" (UID: \"49c23611-783c-4a3f-86d4-46a3617f78f5\") " pod="openshift-must-gather-2sjbp/must-gather-7wtzn"
Jul 01 20:10:43.271398 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.271380    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/49c23611-783c-4a3f-86d4-46a3617f78f5-must-gather-output\") pod \"must-gather-7wtzn\" (UID: \"49c23611-783c-4a3f-86d4-46a3617f78f5\") " pod="openshift-must-gather-2sjbp/must-gather-7wtzn"
Jul 01 20:10:43.279956 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.279936    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwst\" (UniqueName: \"kubernetes.io/projected/49c23611-783c-4a3f-86d4-46a3617f78f5-kube-api-access-dfwst\") pod \"must-gather-7wtzn\" (UID: \"49c23611-783c-4a3f-86d4-46a3617f78f5\") " pod="openshift-must-gather-2sjbp/must-gather-7wtzn"
Jul 01 20:10:43.392704 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.392662    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 20:10:43.433915 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.433887    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2sjbp/must-gather-7wtzn"
Jul 01 20:10:43.472288 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.472258    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad84eab5-b20e-4b2d-829e-11623caf0a1d-proxy-tls\") pod \"ad84eab5-b20e-4b2d-829e-11623caf0a1d\" (UID: \"ad84eab5-b20e-4b2d-829e-11623caf0a1d\") "
Jul 01 20:10:43.472483 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.472310    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad84eab5-b20e-4b2d-829e-11623caf0a1d-openshift-service-ca-bundle\") pod \"ad84eab5-b20e-4b2d-829e-11623caf0a1d\" (UID: \"ad84eab5-b20e-4b2d-829e-11623caf0a1d\") "
Jul 01 20:10:43.472781 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.472753    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad84eab5-b20e-4b2d-829e-11623caf0a1d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ad84eab5-b20e-4b2d-829e-11623caf0a1d" (UID: "ad84eab5-b20e-4b2d-829e-11623caf0a1d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:10:43.474360 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.474335    2586 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad84eab5-b20e-4b2d-829e-11623caf0a1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad84eab5-b20e-4b2d-829e-11623caf0a1d" (UID: "ad84eab5-b20e-4b2d-829e-11623caf0a1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:10:43.562018 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.561993    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2sjbp/must-gather-7wtzn"]
Jul 01 20:10:43.564131 ip-10-0-141-93 kubenswrapper[2586]: W0701 20:10:43.564105    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49c23611_783c_4a3f_86d4_46a3617f78f5.slice/crio-d4798048b4b3245bb935a9ea76b653c15b52f16e9c68f5250150eba85e9bc760 WatchSource:0}: Error finding container d4798048b4b3245bb935a9ea76b653c15b52f16e9c68f5250150eba85e9bc760: Status 404 returned error can't find the container with id d4798048b4b3245bb935a9ea76b653c15b52f16e9c68f5250150eba85e9bc760
Jul 01 20:10:43.565869 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.565853    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 20:10:43.573671 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.573650    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad84eab5-b20e-4b2d-829e-11623caf0a1d-proxy-tls\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 20:10:43.573757 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:43.573678    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad84eab5-b20e-4b2d-829e-11623caf0a1d-openshift-service-ca-bundle\") on node \"ip-10-0-141-93.ec2.internal\" DevicePath \"\""
Jul 01 20:10:44.168771 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:44.168670    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2sjbp/must-gather-7wtzn" event={"ID":"49c23611-783c-4a3f-86d4-46a3617f78f5","Type":"ContainerStarted","Data":"d4798048b4b3245bb935a9ea76b653c15b52f16e9c68f5250150eba85e9bc760"}
Jul 01 20:10:44.169878 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:44.169848    2586 generic.go:356] "Generic (PLEG): container finished" podID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" containerID="9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439" exitCode=0
Jul 01 20:10:44.169999 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:44.169891    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"
Jul 01 20:10:44.169999 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:44.169937    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" event={"ID":"ad84eab5-b20e-4b2d-829e-11623caf0a1d","Type":"ContainerDied","Data":"9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439"}
Jul 01 20:10:44.169999 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:44.169982    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz" event={"ID":"ad84eab5-b20e-4b2d-829e-11623caf0a1d","Type":"ContainerDied","Data":"3cf2e9e9c4e15999ecf9b405ac8a526f7d6ce879c0e5e01d4bd9aa94936e54ad"}
Jul 01 20:10:44.170099 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:44.170007    2586 scope.go:117] "RemoveContainer" containerID="9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439"
Jul 01 20:10:44.177735 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:44.177718    2586 scope.go:117] "RemoveContainer" containerID="9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439"
Jul 01 20:10:44.178019 ip-10-0-141-93 kubenswrapper[2586]: E0701 20:10:44.178002    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439\": container with ID starting with 9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439 not found: ID does not exist" containerID="9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439"
Jul 01 20:10:44.178083 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:44.178026    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439"} err="failed to get container status \"9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439\": rpc error: code = NotFound desc = could not find container \"9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439\": container with ID starting with 9ceb8c450600c74b009dbfdf2eb1f289197d4f6a0d94c76e56bb093002975439 not found: ID does not exist"
Jul 01 20:10:44.185138 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:44.185110    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"]
Jul 01 20:10:44.188776 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:44.188753    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-00068-5f66c94d7d-sd6wz"]
Jul 01 20:10:44.208638 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:44.208305    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:10:45.176728 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:45.176675    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2sjbp/must-gather-7wtzn" event={"ID":"49c23611-783c-4a3f-86d4-46a3617f78f5","Type":"ContainerStarted","Data":"866665f3a0bcf295a3a0a575938045e934f123b423579ef540ef22d5dca8cddc"}
Jul 01 20:10:45.177186 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:45.176744    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2sjbp/must-gather-7wtzn" event={"ID":"49c23611-783c-4a3f-86d4-46a3617f78f5","Type":"ContainerStarted","Data":"d9011e1d9031057ed8643aa368a5ca18fc4833b6dbc6404ca3712e3c837b4d99"}
Jul 01 20:10:45.193244 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:45.193190    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2sjbp/must-gather-7wtzn" podStartSLOduration=1.429557944 podStartE2EDuration="2.193168135s" totalImagesPullingTime="763.610191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 20:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:10:45.19213644 +0000 UTC m=+2959.738385777" watchObservedRunningTime="2026-07-01 20:10:45.193168135 +0000 UTC m=+2959.739417474"
Jul 01 20:10:45.753059 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:45.753028    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5zvjf_55e6ffec-a820-4802-89cb-747770b45bb9/global-pull-secret-syncer/0.log"
Jul 01 20:10:45.885479 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:45.884920    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" path="/var/lib/kubelet/pods/ad84eab5-b20e-4b2d-829e-11623caf0a1d/volumes"
Jul 01 20:10:45.892997 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:45.892968    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log"
Jul 01 20:10:46.022412 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:46.022347    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-93.ec2.internal_4ea467cf493fb8782e204574fc06f1c8/haproxy/0.log"
Jul 01 20:10:48.722013 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:48.721909    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:10:48.822828 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:48.822795    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-56r4t_9dd6796f-9e3d-4c78-93ec-acd06d798d3c/konnectivity-agent/0.log" limit=1024
Jul 01 20:10:50.264536 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:50.264489    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-8kssh_849b87f9-bd2e-41c4-acd7-23fbcea88f8d/kube-state-metrics/0.log"
Jul 01 20:10:50.292136 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:50.292023    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-8kssh_849b87f9-bd2e-41c4-acd7-23fbcea88f8d/kube-rbac-proxy-main/0.log"
Jul 01 20:10:50.321476 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:50.321444    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-8kssh_849b87f9-bd2e-41c4-acd7-23fbcea88f8d/kube-rbac-proxy-self/0.log"
Jul 01 20:10:50.514234 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:50.514192    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8gks2_a1e76d90-156f-4f30-95c7-bfcc96ee0784/node-exporter/0.log"
Jul 01 20:10:50.540734 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:50.540590    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8gks2_a1e76d90-156f-4f30-95c7-bfcc96ee0784/kube-rbac-proxy/0.log"
Jul 01 20:10:50.563935 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:50.563909    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8gks2_a1e76d90-156f-4f30-95c7-bfcc96ee0784/init-textfile/0.log"
Jul 01 20:10:53.442336 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:53.442308    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d77d55d66-dsjjk_3ca503e9-7c14-4560-9ec6-85d11b89e2bc/console/0.log"
Jul 01 20:10:54.150654 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.150607    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"]
Jul 01 20:10:54.151284 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.151258    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" containerName="switch-graph-00068"
Jul 01 20:10:54.151438 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.151405    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" containerName="switch-graph-00068"
Jul 01 20:10:54.151560 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.151501    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad84eab5-b20e-4b2d-829e-11623caf0a1d" containerName="switch-graph-00068"
Jul 01 20:10:54.156529 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.156498    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.166239 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.166205    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"]
Jul 01 20:10:54.268855 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.268816    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-lib-modules\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.269049 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.268892    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-sys\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.269049 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.268922    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-proc\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.269049 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.268960    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrswp\" (UniqueName: \"kubernetes.io/projected/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-kube-api-access-wrswp\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.269049 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.268989    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-podres\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.370368 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.370326    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-lib-modules\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.370534 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.370404    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-sys\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.370534 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.370433    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-proc\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.370534 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.370472    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrswp\" (UniqueName: \"kubernetes.io/projected/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-kube-api-access-wrswp\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.370534 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.370503    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-podres\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.370534 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.370508    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-lib-modules\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.370752 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.370537    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-proc\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.370752 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.370511    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-sys\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.370752 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.370624    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-podres\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.379857 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.379816    2586 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrswp\" (UniqueName: \"kubernetes.io/projected/fa504a4b-6fb3-49a3-bebe-a25d4e67e19f-kube-api-access-wrswp\") pod \"perf-node-gather-daemonset-w4vd7\" (UID: \"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.471348 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.471312    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:54.638868 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.638821    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"]
Jul 01 20:10:54.997719 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:54.997585    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sdr2c_60583eee-9b12-46e0-a65d-d9ca56efc9fb/dns/0.log"
Jul 01 20:10:55.017657 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:55.017632    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sdr2c_60583eee-9b12-46e0-a65d-d9ca56efc9fb/kube-rbac-proxy/0.log"
Jul 01 20:10:55.040957 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:55.040832    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gd8x7_0723c42e-0fc4-4c03-b5e6-83d0af6a9488/dns-node-resolver/0.log"
Jul 01 20:10:55.234095 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:55.234059    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7" event={"ID":"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f","Type":"ContainerStarted","Data":"25e6bd7103d08aafb994f68ec382edb30b6634ffb8a0ebbda400d009700ca84e"}
Jul 01 20:10:55.234285 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:55.234121    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7" event={"ID":"fa504a4b-6fb3-49a3-bebe-a25d4e67e19f","Type":"ContainerStarted","Data":"28facdbdaddd0db7ff723a29bdd1ec6f4ae7c3c1e412e000dc6b94ab80c37a1f"}
Jul 01 20:10:55.234285 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:55.234189    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:10:55.258192 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:55.258070    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7" podStartSLOduration=1.258048445 podStartE2EDuration="1.258048445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:10:55.257986891 +0000 UTC m=+2969.804236228" watchObservedRunningTime="2026-07-01 20:10:55.258048445 +0000 UTC m=+2969.804297782"
Jul 01 20:10:55.638257 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:55.638169    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nrf8m_a0e28d47-acbf-4c6b-ad83-a883ad149185/node-ca/0.log"
Jul 01 20:10:56.866080 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:56.866048    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jjqmq_9d1905ad-8654-45ed-b2d8-2f3cb60cdfd6/serve-healthcheck-canary/0.log"
Jul 01 20:10:57.234495 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:57.234469    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5hm57_88c77ad8-3d00-4d21-8ac6-d6772afccf31/kube-rbac-proxy/0.log"
Jul 01 20:10:57.266488 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:57.266461    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5hm57_88c77ad8-3d00-4d21-8ac6-d6772afccf31/exporter/0.log"
Jul 01 20:10:57.302024 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:57.301995    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5hm57_88c77ad8-3d00-4d21-8ac6-d6772afccf31/extractor/0.log"
Jul 01 20:10:59.590110 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:59.590077    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-bd65f9df8-w6r79_390ef545-bdd8-40c2-b951-0ec703c3c24e/manager/0.log"
Jul 01 20:10:59.609967 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:10:59.609933    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-6zphh_43bc5e3e-2fbb-4775-86b4-1afbdbea5a32/manager/0.log"
Jul 01 20:11:01.249625 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:01.248790    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-w4vd7"
Jul 01 20:11:04.967927 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:04.967870    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-52gwt_908385ef-e1e8-4672-90f9-ef5f5a47f6aa/kube-multus/0.log"
Jul 01 20:11:05.145158 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:05.145133    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b29mt_b5a828de-7dd6-45af-b69c-1837becfe480/kube-multus-additional-cni-plugins/0.log"
Jul 01 20:11:05.166266 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:05.166189    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b29mt_b5a828de-7dd6-45af-b69c-1837becfe480/egress-router-binary-copy/0.log"
Jul 01 20:11:05.187457 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:05.187430    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b29mt_b5a828de-7dd6-45af-b69c-1837becfe480/cni-plugins/0.log"
Jul 01 20:11:05.208878 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:05.208849    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b29mt_b5a828de-7dd6-45af-b69c-1837becfe480/bond-cni-plugin/0.log"
Jul 01 20:11:05.233039 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:05.233008    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b29mt_b5a828de-7dd6-45af-b69c-1837becfe480/routeoverride-cni/0.log"
Jul 01 20:11:05.262111 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:05.262087    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b29mt_b5a828de-7dd6-45af-b69c-1837becfe480/whereabouts-cni-bincopy/0.log"
Jul 01 20:11:05.283061 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:05.283018    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b29mt_b5a828de-7dd6-45af-b69c-1837becfe480/whereabouts-cni/0.log"
Jul 01 20:11:05.570451 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:05.570422    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jbvd8_f41077b2-239b-44f8-be7d-b8b9464b42e9/network-metrics-daemon/0.log"
Jul 01 20:11:05.592617 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:05.592592    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jbvd8_f41077b2-239b-44f8-be7d-b8b9464b42e9/kube-rbac-proxy/0.log"
Jul 01 20:11:06.546095 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:06.546062    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxjj5_d69d7279-1771-42d1-bde5-3dbc8ee7f990/ovn-controller/0.log"
Jul 01 20:11:06.562912 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:06.562886    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxjj5_d69d7279-1771-42d1-bde5-3dbc8ee7f990/ovn-acl-logging/0.log"
Jul 01 20:11:06.584815 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:06.584788    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxjj5_d69d7279-1771-42d1-bde5-3dbc8ee7f990/kube-rbac-proxy-node/0.log"
Jul 01 20:11:06.606081 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:06.606058    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxjj5_d69d7279-1771-42d1-bde5-3dbc8ee7f990/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 20:11:06.627134 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:06.627097    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxjj5_d69d7279-1771-42d1-bde5-3dbc8ee7f990/northd/0.log"
Jul 01 20:11:06.651670 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:06.651644    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxjj5_d69d7279-1771-42d1-bde5-3dbc8ee7f990/nbdb/0.log"
Jul 01 20:11:06.671357 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:06.671332    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxjj5_d69d7279-1771-42d1-bde5-3dbc8ee7f990/sbdb/0.log"
Jul 01 20:11:06.712805 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:06.712775    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxjj5_d69d7279-1771-42d1-bde5-3dbc8ee7f990/ovnkube-controller/0.log"
Jul 01 20:11:07.949019 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:07.948963    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bc752_538fff2a-3657-456b-9158-297890173d5c/network-check-target-container/0.log"
Jul 01 20:11:08.888849 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:08.888820    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-ktjxz_2bc24c26-f688-41bb-ad3e-27d184fe1669/iptables-alerter/0.log"
Jul 01 20:11:09.514718 ip-10-0-141-93 kubenswrapper[2586]: I0701 20:11:09.514670    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-gklml_c775b400-2625-4656-a442-a8069e8bd20e/tuned/0.log"